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ABSTRACT 


The  purpose  of  this  thesis  is  to  analyze  the  probabilistic  cost  model  currently  in 
use  by  NAVSEA  05  C  to  predict  cost  uncertainty  in  naval  vessel  construction  and  to 
develop  a  method  that  better  predicts  the  ultimate  cost  risk.  The  data  used  to  develop  the 
improved  approach  is  collected  from  analysis  of  the  CG(X)  class  ship  by  NAVSEA  05C. 
The  NAVSEA  05  C  cost  risk  factors  are  reviewed  and  analyzed  to  determine  if  different 
factors  are  better  cost  predictors.  The  impact  of  data  elicitation,  the  Money  Allocated  Is 
Money  Spent  (MAIMS)  principle,  and  correlation  effects  are  incorporated  into  the 
research  and  analysis  of  this  thesis.  Data  quality  is  directly  affected  by  data  elicitation 
methods  and  influences  the  choice  of  probability  distribution  used  to  give  the  best 
predictor  of  cost  risk.  MAIMS  and  correlation  effects  are  shown  to  make  a  significant 
impact  to  the  overall  cost  model.  Program  managers  and  analysts  can  readily  implement 
the  enhanced  models  using  commercial  Excel  add-ins,  such  as  Crystal  Ball  or  @Risk,  and 
integrate  them  into  their  current  cost  risk  analysis  and  management  practices  to  better 
mitigate  risk  and  control  project  cost. 
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EXECUTIVE  SUMMARY 


In  order  to  generate  the  funds  to  implement  the  30-year  plan  of  future  ships  and 
eapabilities,  the  Navy  must  explore  different  options  for  eost  savings.  Fundamental  to 
the  sueeess  of  eomplex  projeets,  sueh  as  naval  vessel  eonstruetion,  is  the  ability  to 
eontrol,  manage,  and  eommunieate  the  status  of  the  risk  reduetion  effort  throughout  the 
development  and  produetion  eyeles  (Kujawski  &  Angelis,  2007).  It  is  reeognized  that  the 
Navy  and  the  shipbuilding  industry  need  to  ehange  their  technical  and  business 
shipbuilding  strategies  in  order  to  achieve  the  goal  of  a  future  Fleet  that  balances  both 
capability  and  affordability.  Cost  risk  assessment  and  analysis  is  one  tool  that  can  be 
utilized  to  help  recapitalize  costs  used  in  the  ship  acquisition  and  building  process. 

This  thesis  analyzes  the  probabilistic  cost  model  currently  in  use  by  Naval  Sea 
Systems  Command  Cost  Engineering  and  Industrial  Analysis  Division  (NAVSEA  05C) 
to  predict  cost  uncertainty  in  naval  vessel  construction  and  to  develop  a  method  that 
better  predicts  the  ultimate  cost  risk.  The  NAVSEA  05C’s  cost  analysis  model  for  the 
proposed  new  cruiser,  CG(X),  encompasses  all  aspects  of  cost  for  the  entire  Elect, 
including  inflation  and  profit.  The  data  used  in  the  NAVSEA  model  were  acquired  from 
subject  matter  expert  (SME)  inquiry  using  three-point  estimates  of  high,  most  likely,  and 
low  values.  The  Navy  is  placing  great  emphasis  on  producing  the  best  product  for  each 
dollar  spent.  In  order  to  ensure  the  continued  acquisition  of  CG(X),  it  is  important  that 
realistic  cost  risk  analysis  be  performed  so  that  program  managers  can  make 
informed  decisions. 

The  cost  model  elements  investigated  in  this  thesis  include  data  elicitation 
methods,  probability  distribution  function  (PDE)  choice,  correlation  effects,  and  Money 
Allocated  is  Money  Spent  (MAIMS)  principle  effects.  The  most  significant  impact  is 
seen  with  MAIMS  and  data  elicitation  effects.  PDE  choice  and  correlation  effects  have 
lesser  impact  upon  the  cost  model. 

Methods  of  data  elicitation  are  explored  and  the  use  of  a  direct  fractile  assessment 
(DEA)  is  recommended  for  future  use  (Kujawski,  Alvaro,  and  Edwards,  2004),  although 
the  research  in  this  thesis  did  not  involve  data  acquisition.  To  simulate  the  use  of  a  DEA 


XV 


methodology,  three-parameter  Weibull  distributions  are  employed  to  aeeount  for 
uneertainty  assoeiated  with  SME  estimation  of  data.  A  Weibull  (10%,  50%,  90%) 
distribution  is  used  to  simulate  a  more  optimistic  view  of  the  uncertainty  of  data,  while  a 
Weibull  (20%,  50%,  80%)  distribution  models  a  more  pessimistic  view. 

The  methodology  in  choosing  different  probability  distribution  functions  and  their 
applicability  to  the  model  is  evaluated.  Specifically,  triangular,  lognormal,  and  two 
variations  of  the  three-parameter  Weibull  distribution  are  considered.  Once  enhanced 
models  are  established,  program  managers  can  implement  them  into  their  current  cost 
risk  analysis  practice  to  mitigate  risk  and  control  project  cost. 

Two  types  of  correlation  effects  are  considered  and  modeled  in  this  thesis.  The 
first  is  the  correlation  between  the  components  of  the  radar  suite,  and  the  second  is  the 
correlation  between  all  the  components  of  the  electronics  suite.  The  radar  suite  is  one  of 
the  systems  that  make  up  the  electronics  suite.  The  results  suggest  that  the  correlation 
effects  are  important  for  probability  values  midway  between  the  mean  and  the  extremes, 
but  there  is  little  difference  for  correlation  coefficients  beyond  0.5.  Further  investigations 
are  recommended  to  quantify  correlation  effects. 

MAIMS  modified  probability  distributions  are  used  to  show  the  significance  of 
budget  allocation  levels  (Kujawski,  Alvaro  &  Edwards,  2004).  These  distributions  reflect 
an  empirically  observed  effect,  namely,  that  once  a  budget  is  allocated,  the  project  cost 
will  most  likely  be  at  least  equal  to  the  amount  allocated.  As  the  MAIMS  modification 
value  increases,  the  overall  distribution  cost  rises  with  increasing  probability. 

Credibility  and  realism  are  two  key  cost  risk  assessment  criteria.  The  use  of 
improved  methods,  such  as  those  investigated  in  this  thesis,  are  especially  significant  for 
today’s  Navy  during  a  time  of  budget  hardship.  If  the  Navy’s  plans  for  a 
313-ship  Fleet  are  to  become  a  reality,  the  incorporation  of  cost  risk  analysis  into 
acquisition  and  shipbuilding  management  is  imperative.  Reliable  cost  assessments  can 
help  deliver  projects  on  time,  at  a  lower  cost,  with  a  higher  probability  of  success. 
Effective  training  of  personnel  involved  in  cost  assessment  and  continued  efforts  to 
improve  existing  cost  models  will  help  to  improve  the  Navy’s  current  cost 
estimating  process. 
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I.  INTRODUCTION 


Admiral  Gary  Roughead  stated  in  the  Chief  of  Naval  Operation’s  (CNO) 
Guidanee  for  2007-2008  that: 

We  manage  risk.  We  will  identify,  analyze,  mitigate  and  then 
aeeept  risk,  appreeiating  that  we  must  always  eonsider  the  risks  in 
aggregate  aeross  the  entire  foree.  Zero  risk  is  not  aehievable  nor 
affordable.  We  must  manage  risk  and  move  forward  to  aeeomplish  the 
mission  while  safeguarding  our  people  and  infrastructure 
(Roughhead,  2007). 

Vice  Admiral  K.  M.  McCoy  took  this  a  step  further  in  2008,  in  a  statement  made 
on  assuming  the  position  of  Commander,  Naval  Sea  Systems  Command: 

Our  Common  mission  is  to  develop,  deliver  and  maintain  ships  and 
systems  on  time  and  on  cost  for  the  Navy.  To  build  an  affordable  future 
fleet,  we  will  focus  on  reducing  acquisition  costs,  including  applying  more 
risk-based  decisions  to  specifications  and  requirements 
(McCoy,  2008). 

The  United  States  Navy  is  living  and  functioning  in  an  era  of  ever  expanding 
technology,  more  stringent  requirements,  and  a  growing  need  for  more  ships  and 
resources,  all  while  working  with  a  limited  budget.  These  factors  all  lead  to  inherent  cost 
growth  in  the  projects  that  are  developed  to  provide  the  Fleet  with  the  capabilities  that  it 
needs.  In  order  for  the  United  States  Navy  to  acquire  and  provide  a  full, 
state-of-the-art,  313-ship  Navy  by  2020,  as  stated  in  the  fiscal  year  (FY)  2007  plan 
(Department  of  the  Navy,  2006,  it  is  imperative  that  methods  allowing  full  capitalization 
of  each  dollar  spent  by  the  Navy  are  developed  and  implemented. 

In  February  2006,  the  United  States  Navy  presented  its  FY2007  plan,  which 
outlines  the  objective  of  increasing  the  current  285-ship  Fleet  to  313  ships  by  2020 
(Department  of  the  Navy,  2006).  By  2008,  the  Navy  increased  the  estimate  of  its  annual 
cost  for  the  30-year  plan  by  about  44%  in  real  terms,  but  it  is  still  approximately  7%  less 
than  independent  cost  estimates  conducted  by  the  Congressional  Budget  Office 
(O’Rourke,  2008).  This  increase  in  estimated  cost  poses  a  problem  for  the  overall 
funding  of  the  shipbuilding  strategy  proposed  by  the  Secretary  of  the  Navy.  The 
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credibility  of  the  Navy’s  estimates  and  the  ability  to  fund  its  shipbuilding  plans  have  been 
questioned  by  Congress  and  industry  (Cavas,  2008b). 

In  order  to  adequately  generate  the  funds  to  implement  the  30-year  plan  of  future 
ships  and  eapabilities,  the  Navy  must  explore  different  options  for  eost  savings. 
Fundamental  to  the  sueeess  of  eomplex  projeets,  sueh  as  naval  vessel  eonstruetion,  is  the 
ability  to  eontrol,  manage,  and  eommunieate  the  status  of  the  risk  reduetion  effort 
throughout  the  development  and  produetion  eyeles  (Kujawski  &  Angelis,  2007).  It  is 
reeognized  that  the  Navy  and  shipbuilding  industry  need  to  ehange  their  teehnieal  and 
business  shipbuilding  strategies  in  order  to  aehieve  the  goal  of  a  future  Fleet  that  balanees 
both  eapability  and  affordability.  Cost  risk  assessment  and  analysis  is  one  tool  that  can 
be  utilized  to  help  reeapitalize  eosts  used  in  the  ship  aequisition  and  building  proeess. 

A,  BACKGROUND 

Risk  analysis  is  an  important  eomponent  of  the  eost  analysis  of  new  vessels 
beeause  aetual  eosts  will  always  have  a  probability  of  differing  from  the  estimate. 
Several  reasons  aeeount  for  the  differenee  between  the  estimate  and  aetual  eost,  whieh 
ean  inelude  laek  of  knowledge  about  the  future,  errors  assoeiated  with  assumptions  and 
eost  estimating  equations,  historieal  data  ineonsisteneies,  and  faetors  eonsidered  in 
making  the  estimate.  The  overall  purpose  of  risk  analysis  is  to  quantify  the  potential  for 
error  (Government  Aeeountability  Offiee  (GAO),  2007).  In  the  ease  of  a  eost  estimate  it 
is  the  probability  that  the  aetual  eost  will  exeeed  the  eost  estimate  or  the  budget.  This 
eost  estimate  allows  for  the  assessment  of  risk  of  a  given  program. 

Cost  overruns  and  growth  are  an  enduring  problem  that  is  not  new  to  the  Navy.  A 
1939  inquiry  from  Seeretary  of  the  Navy  Ray  Spear  asks  the  question,  “Why  do  naval 
vessels  eost  so  mueh?”  In  the  answer  to  this  inquiry,  the  reasons  given  inelude  inereased 
progress  in  marine  engineering  and  naval  eonstruetion,  inereased  horsepower  in 
shipbuilding,  improved  quality  of  building  materials,  inflation,  and  the  praetiee  of  paying 
full  priee  for  the  best  you  ean  buy  naturally  inereases  eosts.  Spear  (1939)  states  that, 
“eare  must  be  taken  in  approving  estimates  to  make  sure  that  they  are  reasonable  and  held 
to  in  the  eost  of  produetion.  When  eontraets  are  negotiated  the  question  of  eosts  should 

be  investigated  and  a  detailed  knowledge  of  approximate  eosts  obtained.”  Just  as  it  was 
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recognized  by  the  Secretary  of  the  Navy  in  1939,  it  is  recognized  by  today’s  Navy 
leadership  that  cost  estimation  is  an  integral  part  of  the  ship  acquisition  process. 

The  Defense  Acquisition  Acronyms  and  Terms  (2003)  defines  risk  as: 

A  measure  of  the  inability  to  achieve  program  objectives  within  defined 
cost  and  schedule  constraints.  Risk  is  associated  with  all  aspects  of  the 
program,  e.g.,  threat,  technology,  design  processes,  or  Work  Breakdown 
Structure  (WBS)  elements.  It  has  two  components,  the  probability  of 
failing  to  achieve  a  particular  outcome,  and  the  consequences  of  failing  to 
achieve  that  outcome. 

Risk  analysis  and  management  can  be  used  to  help  program  managers  more 
effectively  make  acquisition  decisions  and  allocate  their  resources  by  allowing  for  a 
better  understanding  of  program  risks.  Risk  management  is  a  continuous  process  in  the 
acquisition  and  development  of  naval  vessels. 

The  Naval  Sea  Systems  Command,  Cost  Engineering  and  Industrial  Analysis 
(NAVSEA  05C)  introduced  Cost  Risk  Analysis  (CRA)  into  the  Navy’s  PR09  Planning, 
Programming,  Budgeting,  and  Execution  System  (PPBES)  to  help  assess  vessel  costs  in 
terms  of  quantifiable  risk.  Cost  Risk  Assessment  is  defined  as  the  process  of  quantifying 
the  uncertainties  associated  with  major  acquisition  programs.  It  therefore  allows  for 
informed  decisions  with  an  estimated  level  of  confidence  (McCarthy,  2008). 

Typical  sources  of  cost  uncertainty  in  naval  vessel  construction  are 
(Deegan  &  Eields,  2007): 

•  Estimating  Methodology.  Cost  Estimation  Relationships  (CER); 
learning/rate/curve  assumptions;  and  cost-reduction  initiatives. 

•  Economic/business  Eactors.  Rates-wages,  overhead.  General  and 
Administrative  (G  &  A),  etc.;  Vendor/supplier  stability;  Inflation  indices; 
Multiyear  assumptions. 

•  Technical  Eactors.  Technology  Readiness  Eevel  (TRL);  design  and 
manufacturing  complexity;  software  complexity;  part  or 
technology  obsolescence. 

•  Schedule  Eactors.  Potential  for  schedule  delays,  how  schedule  risks 
impact  costs. 
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•  Program  Specific  Issues.  Requirements  ehange,  quantity  ehange, 
funding  uneertainty. 

•  Errors  of  Omission  and  Commission.  Failure  to  aeeount  for  rework. 

•  Other  Factors.  Strikes,  acts  of  nature,  accidents. 

One  of  the  key  objectives  of  CRA  is  to  enable  better  risk  management,  which  will 
simultaneously  reduce  program  costs  and  increase  the  probability  of  success.  Cost 
estimating  is  recognized  by  NAVSEA  05  C  as  an  essential  element  of  effective  program 
management,  required  for  realistic  program  planning  and  decision  making.  Risk  analysis 
is  important  because  the  previous  methodology  of  using  point  estimates  is  “precisely 
wrong”  (Deegan,  2007a).  Risk  cannot  be  assessed  with  a  point  estimate,  as  it  represents  a 
single  value  that  serves  as  a  best  guess  for  the  parameter  to  be  defined.  Decision  makers 
may  not  be  able  to  completely  understand  the  influence  of  different  variables  on  cost  with 
the  use  of  a  point  estimate.  Conversely,  the  use  of  risk  analysis  allows  the  decision 
maker  to  utilize  their  acquisition  experience,  while  quantifying  the  qualitative  aspects  of 
acquisition  scenarios. 

Point  estimates  are  not  an  accurate  method  for  predicting  costs  in  shipbuilding 
because  they  do  not  properly  account  for  problems  that  may  be  encountered  in  the 
acquisition  process,  as  described  above.  They  may  be  either  overly  optimistic  or  overly 
pessimistic.  Optimistic  point  estimates  ignore  the  potential  risk  and  uncertainty  in  a 
project,  which  is  necessary  for  management  to  make  informed  decisions.  Immature 
technology,  uncertain  product  design,  schedule  problems,  and  unforeseen  events  all  have 
risk  associated  with  their  end  product.  Risk  analysis  is  necessary  in  order  to  incorporate 
the  effect  of  risk  into  the  overall  cost.  Pessimistic  point  estimates  assume  worst  scenarios 
and  unlikely  high  costs.  Quantitative  risk  analysis  allows  the  cost  estimator  to  assign  a 
realistic  range  of  costs  around  a  point  estimate,  which  provides  decision  makers  with  a 
level  of  confidence  in  achieving  a  credible  cost. 

The  NAVSEA  Cost  Estimating  Handbook  (2005,  p.  3-1)  states: 

Cost  estimators  must  be  proficient  and  aware  of  the  financial  management, 
performance  measurement,  schedule  analysis,  acquisition  management,  as 
well  as  the  technical  aspects  of  a  program  to  support  the  cost  estimating 
process  effectively. 
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The  NAVSEA  Cost  Estimating  Process  is  comprised  of  three  parts,  which  are 
further  divided  into  12  tasks.  The  three  parts  are;  Develop  Approach,  Perform  Estimate, 
and  Brief  Results.  Figure  1  shows  the  breakdown  of  the  12  tasks  within  the  three  parts. 
This  thesis  focuses  on  the  Develop  Approach  and  Perform  Estimate  parts  of  the  cost 
estimating  process. 


Task  1.  Initiate  Estimate  Task  7.  Run  Model  6c  Geoetate  Point  Task-10.  Produetfinal  Esdmaae 

TsukZ  Develop TechniailBueluie  Estimue  Ta^l. 

Tuk  3.  Obtain  WBS  Task  8.  Coodun  Cost  Risk  Anaiyais  &  Task  IZ  Brief  Resula 

Task  4.  P.t.bli.b  A.«.mprioo.  Incorporate  into  Estimate 

Task  5.  Select  Cost  Estimating  Methods  Preliminary  Estimate 

&TooU  Review 

Task  6.  CoUeaDaea 


Part  1  Part  2  Part  3 

Figure  1.  NAVSEA  12-Step  Cost  Estimating  Process  (From;  NAVSEA  Cost 

Estimating  Handbook,  2005) 

This  thesis  addresses  the  first  two  parts  of  the  cost  estimating  process  depicted  in 
Figure  1 — Develop  Approach  and  Perform  Estimate.  Data  collection  is  a  task  within  the 
Develop  Approach  part  of  cost  estimation  and  can  be  regarded  as  the  most  important  part 
of  risk  analysis.  Bad  data  will  produce  bad  results,  regardless  of  the  subsequent  analysis. 
Data  elicitation  is  often  done  ad  hoc;  however,  several  reliable  methods  and  sources  are 
available  for  data  collection.  Data  quality  is  critical  to  the  success  of  the  analysis  and 
plays  a  significant  part  in  the  results  generated  for  cost  estimation.  This  thesis  will 
discuss  improved  methods  for  data  collection  in  order  to  obtain  more  reliable  and 
standardized  data  from  subject  matter  experts  (SMEs). 

Risk  analysts  use  probability  distributions  rather  than  point  estimates  to  represent 
the  possible  outcomes  of  an  event.  There  is  a  significant  difference  between  a  point 
estimate  and  a  distribution,  in  that  the  distribution  provides  the  full  range  of  values  with 
their  associated  probabilities,  while  the  point  estimate  presents  a  single  value.  This 
allows  program  management  to  make  budget  decisions,  based  on  desired  confidence 


5 


levels.  Quality  may  differ,  based  on  the  method  of  colleetion.  Two  methods  eommonly 
used  for  data  eollection  inelude  database  queries  and  interviews  of  SMEs  or  stakeholders 
(Deegan  &  Fields,  2007).  This  thesis  analyzes  the  current  NAVSEA  05C  Cruiser 
(CG(X))  probabilistic  cost  model  including  data  elicitation. 

The  direct  fractile  assessment  (DFA)  method  provides  one  of  the  most  reliable 
and  least  bias-prone  procedures  for  eliciting  uncertain  quantities  from  SMEs  (Kujawski 
et  ah,  2004).  Data  elicitation  from  SMEs  is  innately  uncertain;  three  findings  from 
psychological  experiments  conducted  by  Alpert  and  Raiffia  (1982)  are: 

•  A  systematic  bias  toward  overconfidence  is  common. 

•  Extreme  value  judgment  is  poor. 

•  Maximum  and  minimum  values  are  vague  terms.  What  do  these  terms 
really  mean? 

Based  on  the  findings  of  Alpert  and  Raiffia  (1982),  Kujawski  et  al.  (2004)  propose  the 
following  guidelines  for  data  elicitation: 

•  Ask  SMEs  to  provide  10th,  50th,  and  90th  percentiles  values  for  cost 
elements.  Avoid  extreme  values,  abstract  measures  such  as  the  mean  or 
standard  deviation,  or  specific  distribution  functions.  Allow  for  discussion 
and  education  of  the  SME  in  terms  of  bias  when  giving  data  figures. 

•  Calibrate  each  set  of  percentiles  to  reflect  individual  and  project  specific 
considerations,  both  pessimistic  and  optimistic.  For  estimates  that  might 
be  overly  optimistic,  a  cost  analyst  might  choose  to  shift  a  90th  percentile 
value  to  perhaps  80th  or  75th  percentiles. 

Tasks  involved  in  the  Performing  Estimate  depicted  in  Figure  1  are  running  the 
model  and  generating  a  point  estimate  or  probability  distribution,  conducting  a  cost  risk 
analysis,  and  conducting  a  preliminary  estimate  review. 

Traditionally,  triangular  distributions  have  been  used  in  cost  estimation  models 
because  of  the  simplicity  in  entering  the  required  data.  The  triangular  distribution 
requires  minimum  or  low,  most  likely,  and  high  or  maximum  values.  Other  commonly 
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used  distributions  include  normal,  lognormal,  and  uniform.  Table  1  lists  eight  of  the 
most  common  probability  distributions  used  for  cost  estimation  and  uncertainty  analysis. 
This  thesis  investigates  different  methods  for  data  elicitation  and  selecting  appropriate 
distributions.  The  effects  of  using  different  distributions  on  cost  risk  are  evaluated 
and  identified. 


Distribution 

Description 

Shape  Typical  application 

Lognormal 

A  continuous  distribution  positively  P: 

skewed  with  a  limitless  upper  bound 
and  known  lower  bound;  skewed  to 
the  right  to  reflect  the  tendency  toward 
higher  cost. 

To  characterize  uncertainty  in 
nonlinear  cost  estimating 
relationships. 

Values 

Normal  Used  for  outcomes  likely  to  occur  on 

either  side  of  the  average  value; 
symmetric  and  continuous,  allowing 
for  negative  costs  and  durations.  In  a 
normal  distribution,  about  68%  of  the 
values  fall  within  one  standard 
deviation  of  the  mean. 

Beta  Similar  to  normal  distribution  but  does 

not  allow  for  negative  cost  or  duration, 
this  continuous  distribution  can  be 
symmetric  or  skewed. 


To  assess  uncertainty  with 
,  ■  cost  estimating  methods;  the 

standard  deviation  or  standard 
error  of  the  estimate  is  used  to 
detemnine  dispersion. 


p^r:  To  Capture  outcomes  biased 

toward  the  tail  ends  of  a  range; 
often  used  with  engineering 
data  or  analogy  estimates. 

Value  ■ 


Triangular  Characterized  by  three  points — most 
likely,  pessimistic,  and  optimistic 
values — can  be  skewed  or  symmetric 
and  is  easy  to  understand  because  it 
is  intuitive.  One  drawback  is  the 
absoluteness  of  the  end  points. 


•y  ,  To  express  technical 

uncertainty,  because  it  works 
for  any  system  architecture  or 
design;  also  used  to  determine 
schedule  uncertainty. 


Uniform  Has  no  peaks  because  all  values,  ^  With  engineering  data  or 

including  highest  and  lowest  possible  analogy  estimates, 

values,  are  equally  likely. 


VElues 


Weibull  Versatile,  able  to  take  on  the 

characteristics  of  other  distributions, 
based  on  the  value  of  the  shape 
parameter  “b” — e.g,,  Rayleigh  and 
exponential  distributions  can  be 
derived  from  it.* 


In  life  data  and  reliability 
analysis  because  it  can  mimic 
other  distributions  and  its 
objective  relationship  to 
reliability  modeling. 

Valuef- 


Source:  DOD,  NASA,  SCEA,  and  Industry, 

TTie  Rayleigh  and  exponential  distributions  are  a  class  of  continuous  probability  distribution. 

Table  1 .  Common  Probability  Distributions  used  in  cost  estimating  uncertainty  analysis 

(From;  GAO  Cost  Assessment  Guide,  2007) 

The  Money  Allocated  is  Money  Spent  (MAIMS)  principle  is  based  on 
Parkinson’s  Law,  where  “Work  expands  to  fill  the  time  allotted”  and  “padding  schedule 
estimates  directly  contribute  to  cost  overruns”  (Augustine,  1997).  In  other  words,  it 
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suggests  that  there  will  be  no  eost  underruns,  and  that  the  project  will  come  in  at  or  above 
the  cost  to  which  it  is  funded.  Implementing  the  MAIMS  principle  in  Monte  Carlo 
simulations  modifies  the  basic  probability  distribution  functions  (PDF)  by  setting  any 
value  less  than  the  money  allocation  point  equal  to  that  money  allocation  value.  There 
will  be  no  costs  associated  with  a  value  less  than  this  money  allocation  point.  Utilizing 
the  MAIMS  principle,  the  PDFs  are  modified  to  include  a  spike  or  delta  function  at  an 
arbitrary  point,  which  is  assumed  to  be  the  “money  allocation  point,”  corresponding  to 
the  dollar  amount  allocated  to  the  program  manager  for  the  project  and/or  project 
cost  elements. 

Correlation  effects  between  elements  are  analyzed.  Correlation  accounts  for 
interrelationships  between  cost  elements.  Data  elements  can  either  be  negatively, 
neutrally,  or  positively  correlated  and  can  either  exist  among  cost  elements  within  a 
subsystem  or  between  elements  in  different  subsystems.  For  example,  take  into 
consideration  the  elements  of  a  ship.  Positive  correlation  arise  when  increases  in  weight, 
size,  and  number  of  weapons  systems  onboard  result  in  an  increase  in  acquisition  and 
shipbuilding  costs.  An  increase  in  the  complexity  of  a  weapon  system  further  forces  an 
increase  in  cost  of  other  systems  such  as  power,  cooling,  control.  Analysis  would  be 
greatly  simplified  if  analysts  could  assume  that  all  elements  are  independent,  or  that  all 
elements  are  dependent.  Since  neither  statement  is  true,  correct  correlation  between 
elements  is  necessary  to  provide  the  most  accurate  representation  of  cost. 

Many  software  programs  are  available  for  cost  risk  analysis.  This  thesis  uses 
Crystal  Ball®  as  an  add-in  to  Microsoft  Excel®,  because  of  its  ease  of  use  and  because  it 
is  the  current  program  used  by  NAVSEA  05C.  Crystal  Ball®  generates  the  Monte  Carlo 
simulations  that  become  the  backbone  of  the  cost  risk  analysis.  A  Monte  Carlo 
simulation  calculates  multiple  scenarios  of  a  model  by  repetitively  sampling  values  from 
the  input  variable  distributions  for  each  uncertain  variable  and  then  calculates  the  result. 
Depending  on  the  complexity  of  the  model,  the  analysis  may  take  only  a  few  moments  or 
hours  (Wittwer,  2004).  The  resulting  cost  distributions  from  Crystal  Ball®  provide  the 
decision  maker  with  powerful  cost  risk  information. 


8 


A  program  built  on  a  solid  foundation  of  accurate  cost  estimating  that  effeetively 
eonsiders  risks,  eombined  with  strong  systems  engineering  and  program  management, 
gives  the  program  a  greater  ehanee  of  success. 

B,  PURPOSE 

The  purpose  of  this  thesis  is  to  analyze  the  probabilistic  cost  analysis  approach 
that  NAVSEA’s  Cost  Engineering  and  Industrial  Analysis  division  (SEA  05C)  eurrently 
uses  to  prediet  new  naval  vessel  eonstruetion  eosts  and  to  develop  a  method  that  better 
predicts  the  ultimate  eost  risk.  This  thesis  uses  data  colleeted  from  analysis  of  the  CG(X) 
elass  ship  eost  model.  The  model  used  to  determine  eost  is  reviewed  and  analyzed  to 
determine  what  faetors  should  be  eonsidered  to  produee  more  realistie  eost  estimates. 

C,  RESEARCH  QUESTIONS 

Several  questions  are  addressed  in  this  thesis.  They  inelude: 

•  What  is  the  realism  of  the  eost  models  currently  in  use  by  NAVSEA  05  C 
for  the  CG(X)  elass  ship? 

•  How  does  data  elieitation  impaet  the  eost  predietion? 

•  How  does  the  ehoiee  of  distributions  in  the  model  affect  the  predictions  of 
the  eost  outeome? 

•  What  is  the  impaet  of  the  MAIMS  prineiple  on  the  eost 
probability  distributions? 

•  How  ean  the  current  CG(X)  model  be  improved  to  provide  more  aecurate 
models  to  predict  cost  and  risk? 

D,  BENEFITS  OF  STUDY 

The  benefit  of  this  study  is  to  provide  an  independent  analysis  of  the  eost  model 
developed  by  NAVSEA  05  C  that  is  eurrently  being  used  in  their  probabilistie  eost 
analysis.  This  analysis  will  present  alternatives  that  may  be  used  to  improve  the  eurrent 
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approach,  to  more  accurately  predict  the  cost  of  a  naval  vessel  and  the  risks  associated 
with  its  development.  The  main  focus  is  on  data  elicitation,  choice  of  probability 
distributions,  the  MAIMS  principle,  and  correlations.  With  the  proposed  cost  analysis 
approach,  the  management  involved  in  the  development  and  construction  of  naval  vessels 
will  have  a  better  tool  to  reduce  program  costs,  while  increasing  the  probability  of 
success  for  each  project. 

E,  SCOPE  AND  METHODOLOGY 

Because  of  the  complexity  of  the  NAVSEA  05C  CG(X)  model,  this  thesis  focuses 
on  the  analysis  of  the  electronics  portion.  The  electronics  suite  is  comprised  of  nine 
different  systems  and  related  components.  From  this  analysis,  a  model  is  developed  to 
provide  a  more  effective  method  to  assess  cost  risk.  Crystal  Ball®  software  is  used  to 
provide  analysis  with  the  use  of  Monte  Carlo  simulations  and  charts  to  help  quantify  the 
importance  of  different  risk  factors  and  their  link  to  the  overall  cost  risk. 

A  literature  review  of  risk  assessment,  project  risk  management,  cost  analysis,  and 
cost  modeling  is  conducted  in  Chapter  II.  Data  is  then  analyzed  in  Chapter  III,  to 
determine  how  the  NAVSEA  CG(X)  model  was  developed,  what  the  assumptions  were, 
what  distributions  were  used,  and  if  correlation  effects  were  used.  In  Chapter  IV, 
Crystal  Ball®  software  is  used  to  conduct  Monte  Carlo  simulations  on  modified  models  to 
determine  if  there  was  any  significant  change  in  the  cost  based  on  proposed  assumptions. 
In  these  simulations,  experiments  with  different  distributions,  correlation  effects,  and 
truncation  of  the  distributions  to  simulate  the  MAIMS  principle  are  conducted.  Finally, 
Chapter  V  summarizes  the  proposed  cost  risk  analysis  improvements  that  were 
investigated  in  this  thesis. 
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II.  LITERATURE  REVIEW 


A,  INTRODUCTION 

Cost  estimation  is  a  process  that  has  been  more  closely  scrutinized  in  recent 
history  because  of  the  increasing  trouble  with  cost  overruns  by  major  programs.  The 
Littoral  Combat  Ship  (LCS)  is  an  example  of  gross  cost  overruns.  The  2008 
United  States  Government  and  Accountability  Office  (GAO)  review,  in  March  2008, 
revealed  that  the  Navy  was  expecting  the  first  two  LCSs  to  exceed  their  budgets  by  more 
than  100  percent,  while  delivery  of  the  first  ship  would  be  at  least  18  months  later  than 
originally  projected.  These  cost  and  schedule  overruns  led  to  the  cancelled  construction 
of  the  third  and  fourth  LCSs.  Funds  originally  allocated  for  construction  of  the  fifth  and 
sixth  LCSs  were  allocated  to  pay  for  the  extensive  cost  growth  experienced  by  LCSs  one 
and  two  (Littoral  Combat  Ship  Program,  2008).  Secretary  of  the  Navy  Donald  Winter 
states:  “  ...  we  recognize  that  active  oversight  and  strict  cost  controls  in  the  early  years 
are  necessary  to  ensuring  we  can  deliver  these  ships  to  the  fleet  over  the  long  term” 
(Scully,  2007).  Inability  to  forecast  the  required  cost  for  future  ships  and  programs  will 
not  allow  the  Navy  to  reach  its  goal  of  a  313-ship  Fleet  by  2020. 

Risk  comes  in  various  forms  in  the  business  of  shipbuilding.  Schedule, 
technology,  source  selection,  and  requirements  are  only  a  few  of  the  factors  that  are 
associated  with  uncertainty  and  risk.  The  following  statements,  somewhat  sarcastically, 
but  with  much  insight,  sum  up  potential  impacts  associated  with  risk  in  a  project: 

•  “Any  task  can  be  completed  in  only  one -third  more  time  than  is  currently 
estimated.”  -  Augustine’s  Law  of  Unmitigated  Optimism,  Law  Number 
XXIII  (Augustine,  1997). 

•  “The  only  thing  more  costly  than  stretching  the  schedule  of  an  established 
project  is  accelerating  it,  which  is  itself  the  most  costly  action  known  to 
man.”  -  Augustine’s  Law  of  Economic  Unipolarity,  Law  Number  XXIV 
(Augustine,  1997). 
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•  “The  process  of  competitively  selecting  contractors  to  perform  work  is 
based  on  a  system  of  rewards  and  penalties,  all  distributed  randomly.”  - 
Augustine’s  Law  of  the  Phoenix,  Law  Number  XXXIV  (Augustine,  1997). 

•  “Ninety  percent  of  the  time  things  will  turn  out  worse  than  you  expect. 
The  other  10  percent  of  the  time  you  had  no  right  to  expect  so  much.”  - 
Augustine’s  Law  of  Apocalyptic  Costing,  Law  Number  XXXVII 
(Augustine,  1997). 

•  “One  should  expect  that  the  expected  can  be  prevented,  but  the  unexpected 
should  have  been  expected.”  -  Augustine’s  Law  of  Amplification  of 
Agony,  Law  Number  XLV  (Augustine,  1997). 

•  “The  sooner  you  start  to  fall  behind,  the  more  time  you  will  have  to  catch 
up”  (Augustine,  1997). 

B,  HISTORY  OF  COST  RISK  ASSESSMENT 

Gambling  has  been  a  human  pastime  for  millennia  and  has  been  practiced  at  all 
levels  of  societies.  Evidence  of  gambling  can  be  found  in  Greek  mythology,  in  which  a 
game  of  “craps”  was  used  to  explain  what  we  commonly  refer  to  as  the  Big  Bang.  In 
Greek  mythology,  Zeus,  Poseidon,  and  Hades  rolled  dice  for  parts  of  the  universe.  Zeus 
won  the  heavens,  Poseidon  the  seas,  and  Hades  became  the  master  of  the  underworld 
(Bernstein,  1998). 

Modern-day  risk  analysis  has  roots  in  the  Hindu-Arabic  numbering  system  as 
early  as  seven  to  eight  hundred  years  ago.  The  new  numbering  system  allowed  writing  to 
take  the  place  of  movable  counters  in  making  calculations.  The  act  of  being  able  to  write 
calculations  opened  mathematics  up  to  a  new  level.  Risk  analysis  became  a  more  serious 
topic  during  the  Renaissance,  in  a  time  when  people  started  to  break  free  from  past 
constraints  and  old  beliefs  to  new  challenges.  In  1654,  Chevalier  de  Mere  challenged 
Blaise  Pascal  to  answer  a  question  that  had  been  posed  by  Luca  Paccioli  over  two 
hundred  years  prior.  The  problem  was  to  determine  how  to  divide  the  stakes  of  an 
incomplete  game  of  chance  between  two  different  players  when  one  of  the  players  is 
ahead.  In  seeking  out  his  answer,  Pascal  sought  the  help  of  Pierre  de  Fermat.  Together 
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they  came  up  with  a  solution  to  the  Paccioli  puzzle.  This  revelation  led  to  the  theory  of 
probability,  the  heart  of  the  concept  of  risk.  Through  this  discovery,  people  could  now 
make  decisions  and  use  numbers  to  forecast  events  in  the  future  (Bernstein,  1998).  In 
1730,  Abraham  de  Moivre  discussed  the  structure  of  the  normal  distribution  and 
developed  the  standard  deviation  (Bernstein,  1998).  Both  of  these  concepts  are  important 
components  of  modern  day  methods  for  conducting  risk  assessment. 

Historically,  cost  growth  has  been  associated  with  many  large  capital  projects, 
especially  those  that  have  long  construction  periods.  The  trend  has  been  that  larger 
budgets  and  longer  build  times  lead  to  greater  risk  and  uncertainty.  Figure  2  shows  a 
history  of  cost  growth  over  the  last  700  years.  This  chart  shows  different  reasons  for 
which  each  of  these  major  projects  came  in  under,  over,  or  right  on  cost  estimates.  An 
interesting  note  is  that  the  Eiffel  Tower  project  builder  was  guaranteed  the  first  20  years 
of  revenue  (Deegan,  2007b).  This  fact  may  have  helped  to  influence  the  overall  cost  of 
the  project  to  come  in  under  budget.  All  of  the  causes  listed  in  Figure  2  can  be  applied  as 
reasons  for  problems  with  the  accuracy  of  cost  estimates  in  current  shipbuilding  projects. 
Being  able  to  quantify  cost  variability  and  uncertainty  is  one  way  to  help  alleviate  major 
cost  overruns. 
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Figure  2.  Historical  Cost  Growth:  Fast  700  Years 

(From:  Deegan,  2007b) 
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C.  CLASSICAL  COST  ESTIMATION  AND  CONTINGENCY  PLANNING 

Cost  estimation  is  an  analytical  effort  directed  at  calculating  and  predicting  the 
cost  of  a  system  that  involves  several  techniques  such  as  data  collection  and  analysis  and 
risk  analysis.  It  is  a  quantitative  assessment  of  the  most  likely  costs  required  to  complete 
a  project  and  always  involves  amounts  of  uncertainty  and  risk.  The  validity  and 
significance  of  a  cost  estimate  relies  on  the  experience  and  judgment  of  those  assigned 
with  making  the  decisions  involved  in  the  estimate.  There  is  no  one  correct  answer  when 
making  a  cost  estimate  as  several  factors  must  be  taken  into  account  in  order  to  provide 
the  best  estimate  for  the  given  situation.  Associated  costs  may  include  labor,  material, 
equipment,  inflation,  and  services.  Cost  estimators  interact  with  many  different 
personnel  while  collecting  data  for  analysis.  Consultation  with  SMEs,  engineers, 
schedulers,  accountants,  statisticians,  and  mathematicians  may  occur  in  the  cost 
estimation  process  (Project  Manager’s  Body  of  Knowledge  (PMBOK),  2004). 

Classical  cost  estimation  techniques  involve  using  point  estimates  as  the  best 
value  while  also  assigning  a  project  contingency  as  a  percentage  of  the  total  cost.  This 
methodology  is  changing  to  the  practice  of  using  probability  distributions  to  model  cost 
without  contingencies.  Cost  contingencies  have  been  associated  with  overestimating 
costs  to  account  for  the  contingency  allowances.  Program  managers  are  responsible  for 
the  budget  management  of  their  projects  and  can  benefit  from  using  cost  estimation 
techniques  for  use  in  budget  planning  and  execution  (Portny,  Mantel,  Meredity,  Shafer,  & 
Sutton,  2008). 

D,  PROBABILISTIC  COST  RISK  ANALYSIS 

Modern  cost  uncertainty  analysis  started  in  a  field  known  as  military  systems 
analysis,  which  was  founded  in  the  1950s  at  the  RAND  Corporation.  The  military 
systems  analysis  branch  evolved  after  World  War  II  and  became  a  tool  to  aid  defense 
leadership  with  decisions  for  force  structure,  composition,  and  theaters  of  operation. 
Naturally,  cost  analysis  became  an  important  part  of  the  analysis  models  and  decision 
process.  Early  cost  uncertainty  analysis  focused  on  the  sources,  scope,  and  types  of 
uncertainties  that  influenced  the  cost  of  future  technologies.  From  1955  to  1962,  the 
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technical  papers  regarding  cost  uncertainty  did  not  focus  on  developing  formal 
methodologies  to  quantify  cost  uncertainty  (Garvey,  2000).  In  the  1960s,  a  new  focus  on 
cost  analysis  had  begun  and  forms  the  basis  for  today’s  cost  estimation  methods. 

Cost  estimation  uncertainties  are  present  in  many  elements  of  cost  analysis.  They 
originate  from  inaccuracies  in  models,  misinterpretation  of  the  models,  misused  cost 
analysis  methodologies,  economic  uncertainties,  requirements  change,  and  system 
definition  uncertainties.  Cost  risk  analysis  is  the  statistical  treatment  of  the  cost 
estimating  process  that  considers  the  elements  of  the  work  breakdown  structure  (WBS) 
and  total  system  costs  as  random  variables.  Cost  analysis  indicates  that  there  is  always 
some  degree  of  risk  that  a  system  may  not  be  delivered  or  meet  required  objectives  at  a 
certain  funding  level.  One  of  the  goals  of  cost  analysis  is  to  be  able  to  assign  a  degree  of 
confidence  to  any  particular  estimate  of  a  system  cost.  Another  factor  to  consider  when 
conducting  cost  risk  assessment  is  to  analyze  what  funding  level  percentage  is 
appropriate  for  the  project. 

Most  cost  estimation  is  based  on  historical  data  obtained  from  SMEs  and 
databases  from  previously  built  systems.  Basing  a  cost  estimate  on  past  systems  can  be 
tricky  and  complicated.  Cost  analysts  have  been  expected  to  provide  a  best  estimate  of 
cost  for  various  options  at  milestones  and  decision  points  in  the  acquisition  of  major 
weapons  programs.  What  precisely  is  the  best  estimate?  Definitions  of  the  term  “best 
estimate”  vary  with  different  projects,  and  with  each  analyst  and  program  manager.  The 
best  estimate  may  be  the  most  likely  cost,  the  median,  or  the  average  cost.  The  term 
“best  estimate”  implies  that  other  less  likely  solutions  may  exist. 

In  probabilistic  or  risk  cost  estimation,  costs  of  each  WBS  element  are  modeled 
with  probability  distributions  and  then  correlations  among  these  elements  are  estimated. 
The  result  of  the  correlated  distributions  is  statistically  summed  by  using  a  simulation 
such  as  Monte  Carlo.  The  simulation  results  in  the  total  system  cost  represented  by  a 
probability  distribution.  Estimates  of  statistical  parameters,  such  as  the  median  or  other 
pertinent  values,  can  be  extracted  from  the  final  probability  distribution.  These  values 
are  meaningful  to  program  managers  and  decision  makers  when  establishing 
program  budgets. 
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Cost  estimation  can  also  be  used  as  a  decision  aid  when  program  managers  need 
to  make  a  decision  about  the  appropriate  funding  level  for  a  project.  Figure  3  shows  the 
impact  of  either  budgeting  with  a  value  less  or  greater  than  the  mean.  Both  decisions 
have  possible  positive  and  negative  outcomes.  It  is  important  for  the  program  manager 
and  budget  decision  makers  to  have  the  most  complete  understanding  of  the  cost  analysis 
and  consequence  to  make  the  best  budget  decisions. 


Figure  3.  Business  Rules  to  Consider:  Choosing  an  acceptable  cost  risk  point  requires 

an  understanding  of  consequence  (From:  Deegan,  2007b). 


Figure  4  is  a  flowchart  that  represents  the  basic  steps  in  estimating  probable 
systems  cost  (Book,  2001).  This  thesis  focuses  on  the  “Probability  Distributions, 
Perform  Monte  Carlo  Simulations,  Cumulative  Distribution,  and  Read  Off  Cost 
Percentiles”  blocks  of  this  process  within  the  Cost-Risk  Analysis  branch. 
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Figure  4.  Basic  Steps  in  Estimating  Probable  Systems  Cost  (From;  Book,  2001) 


E.  CHAPTER  SUMMARY 

There  is  an  ongoing  major  shift  in  research  and  development  and  complex 
engineering  projects  from  deterministic  to  probabilistic  approaches.  Probabilistic  cost 
risk  analysis  provides  the  proper  framework  for  handling  the  many  different  factors  that 
contribute  to  cost  uncertainties.  It  is  an  important  tool  for  program  managers  to  use  in 
making  better  budget  decisions.  This  is  especially  significant  for  the  today’s  Navy  during 
a  time  of  budget  hardship.  In  order  for  the  Navy’s  plans  for  a  313-ship  Fleet  to  become  a 
reality,  the  incorporation  of  cost  risk  analysis  into  the  aspects  of  acquisition  and 
shipbuilding  is  imperative.  Cost  assessment  can  help  to  deliver  projects  on  time,  at  a 
lower  cost,  with  a  higher  probability  of  success.  Effective  training  of  personnel  involved 
in  cost  assessment,  and  continued  efforts  to  improve  on  existing  models  will  help  to  more 
fully  integrate  this  way  of  doing  business  into  the  current  cost  estimating  process. 
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III.  REVIEW  OF  THE  NAVSEA  05C  (COST  ENGINEERING  AND 
INDUSTRIAL  ANALYSIS  DIVISION)  COST  RISK  MODEL 


A,  INTRODUCTION 

The  cost  model  used  for  analysis  in  this  thesis  was  obtained  from  NAVSEA  05  C. 
Mr.  Chris  Deegan,  the  former  Director  of  the  Naval  Sea  Systems  Command  Cost 
Engineering  and  Industrial  Analysis  Division,  and  Mr.  Aaron  Ratliff,  his  CG(X)  analyst, 
provided  the  model.  The  model  contains  information  required  to  determine  the  cost  of  an 
entire  CG(X)  ship  including  economic  factors  such  as  inflation,  profit,  and  learning 
curves  for  the  complete  CG(X)  fleet.  Because  of  the  complexity  of  the  model,  this  thesis 
focuses  on  the  electronics  cost  portion  and  the  effects  of  data  elicitation,  probability 
distribution  choice,  correlation  effects,  and  the  MAIMS  principle  on  the  probabilistic 
costs  associated  with  technology,  design,  engineering,  production,  testing,  and  integration 
risks  for  the  first  vessel. 

B,  REVIEW  OF  CG(X)  SHIP  CLASS  AND  MODELS 

The  CG(X)  ship  is  designed  to  be  the  future  Fleet’s  replacement  for  the 
Ticonderoga  Class  AEGIS  Cruisers.  New  technologies  being  advocated  and  presented  in 
CG(X)  include  a  stealthier  hull  form,  improved  missiles,  integrated  power  systems,  and 
advanced  radar  systems  (Navy  of  the  Future,  2008).  To  lessen  costs,  the  CG(X)  was  to 
share  many  common  features  with  DDG  1000.  As  of  July  2008,  only  two  of  the 
DDG  1000  ships  will  be  built  due  to  funding  constraints.  This  is  a  significant  reduction 
from  the  original  32-ship  DDG  1000  plan  of  the  1990s  (Cavas,  2008a).  These  same 
funding  constraints  place  an  important  emphasis  on  effective  cost  and  risk  modeling  for 
CG(X),  to  help  ensure  that  its  acquisition  is  not  put  in  jeopardy  due  to  unforeseen 
cost  issues. 

The  NAVSEA  CG(X)  cost  model  uses  historical  data  from  DDG  1000  expressed 
in  FY2005  dollars.  It  comprises  63  separate  worksheets  within  the  main  Excel® 
workbook.  The  data  used  in  this  model  consists  of  inputs  from  SME  inquiry,  historical 
data  from  past  shipbuilding  projects,  historical  data  from  shipyard  figures,  and  historical 
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rate  data.  The  NAVSEA  05C  CG(X)  eost  model  takes  into  aeeount  all  faetors  involved 
in  cost  for  every  aspect  of  the  ship  including  labor  rates,  material  cost,  overhead  cost, 
planning  cost,  production  cost,  engineering  cost,  construction  cost,  change  orders, 
electronics  cost,  hull,  mechanical  and  engineering  costs,  and  ordnance.  Figure  5  is  the 
Cost  Placemat  of  the  NAVSEA  05C  CG(X)  model,  which  shows  how  the  cost  breakdown 
is  established. 
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Figure  5.  NAVSEA  05C  CG(X)  Cost  Model;  Cost  Placemat  showing  cost  breakdown. 


Figure  6  shows  a  section  of  the  Mission  Systems  Risk  Assessment  worksheet  of 
the  NAVSEA  05C  CG(X)  Excel®  workbook.  This  worksheet  shows  the  different  cost 
elements  of  the  WBS  for  the  electronics  suite  of  CG(X)  as  well  as  the  WBS  elements  for 
the  ordnance  suite.  The  elements  of  the  ordnance  suite  are  not  considered  in  this  thesis. 
This  figure  is  a  snapshot  of  a  single  step  of  the  Monte  Carlo  simulation.  Note  that 
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Xl(Rec)  was  generated  as  part  of  the  thesis  using  the  Excel®  spreadsheets  provided  by 
NAVSEA  05C.  The  column  “XI  (Rec)”  represents  the  random  variables  corresponding 
to  the  recurring  costs  of  the  first  ship  of  the  class.  It  corresponds  to  the  probabilistic 
costs,  which  include  uncertainty  information.  The  values  for  the  XI  costs  are  calculated 
using  the  subsystem  cost  values  provided  by  the  Input  Template  worksheet  shown  in 
Eigure  7.  The  column  showing  nonrecurring  engineering  (NRE)  costs  are  costs  for  only 
the  lead  ship  of  the  class.  The  values  in  the  “most  likely”  column  are  point  estimates 
with  the  “low”  and  “high”  values  as  the  upper  and  lower  cost  bounds  of  the  individual 
electronics  suite  elements. 

The  Naval  Surface  Warfare  Center  (NSWC),  Dahlgren,  Virginia,  provided  the 
low,  most  likely,  and  high  values  for  the  system  and  component  costs.  These  values 
provide  the  basic  input  for  this  thesis.  The  electronics  values  and  how  they  are 
determined  are  the  focus  of  this  thesis. 
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Figure  6.  NAVSEA  05C  CG(X)  Model  -  Section  of  Mission  Systems  Risk  Assessment 
worksheet  depicting  WBS  Ordnance  and  Electronics  Suite  elements.  It  captures  a 
snapshot  of  a  single  step  of  Monte  Carlo  simulation  run  using  Crystal  Ball®. 
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Figure  7  shows  a  section  of  the  Input  Template  worksheet  from  the  NAVSEA 
05C  CG(X)  model.  As  indicated  by  the  “Distribution  Used”  column,  NAVSEA  05C  uses 
a  lognormal  distribution  for  each  cost  component  in  the  electronics  suite.  It  then 
characterizes  each  lognormal  distribution  in  terms  of  the  mean  and  standard  deviation 
values  given  in  the  Risk  Ranges  Mission  Systems  worksheet,  a  section  of  which  is  shown 
in  Eigure  8.  Analysis  of  the  provided  data  appears  to  indicate  that  NAVSEA  05  C 
computes  the  mean  and  standard  deviation  values  treating  the  “low,”  “most  likely,”  and 
“high“  values  as  a  discrete,  rather  than  a  triangular,  distribution.  This  calculation  of  the 
standard  deviation  and  mean  for  the  lognormal  distribution  has  not  been  confirmed  with 
NAVSEA  05  C.  This  approximation  provides  values  significantly  different  from  those 
based  on  a  triangular  distribution,  as  is  typically  done  in  today’s  de  facto  probabilistic 
cost  analysis.  NAVSEA  05C  introduces  an  additional  approximation  and  treats  the  radar 
suite  as  a  single  composite  cost,  rather  than  in  terms  of  its  individual  cost  elements. 
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Eigure  7.  NAVSEA  05C  CG(X)  Model-  Section  of  the  Input  Template  worksheet 

depicting  the  systems  analyzed  in  this  thesis. 
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Figure  8.  NAVSEA  05C  CG(X)  Model  -  Risk  Ranges  Mission  Systems  worksheet. 
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C.  DATA  SELECTED  FOR  FURTHER  ANALYSIS 

This  thesis  uses  the  low,  most  likely,  and  high  data  values,  which  are  provided  by 
the  Mission  Systems  Risk  Analysis  worksheet  shown  in  Figure  6.  The  Mission  Systems 
Risk  Analysis  worksheet  is  one  of  the  63  worksheets  of  the  CG(X)  model.  The  Xl(Rec) 
value  is  derived  from  data  contained  in  other  worksheets  in  the  model.  The  details  of  the 
methodology  of  how  the  Xl(Rec)  value  is  developed  is  outside  the  scope  of  this  thesis. 
This  thesis  then  evaluates  the  impact  of  different  distributions,  data  elicitation, 
correlations,  and  the  MAIMS  principle  on  the  estimated  cost. 

The  thesis  investigates  the  following  probability  distributions: 

•  Triangular  distributions  using  specified  high,  most  likely,  and  low  values. 

•  Lognormal  using  the  mean  and  standard  values  from  the 
triangular  distributions. 

•  Two  sets  of  Weibull  distributions  using  the  triangular  distribution 
parameters  as  different  percentiles  to  reflect  data  elicitation. 

•  One  set  of  Weibull  distributions  modified  to  account  for  the 
MAIMS  principle. 

It  also  investigates  correlation  effects  using  a  two-parameter  model  (Kujawski 
et  ah,  2004). 

A  discussion  of  data  elicitation  practices  and  methods  is  presented,  and  its  impact 
on  the  quality  of  data  is  explored.  Data  elicitation  can  directly  affect  the  choice  of 
distribution  used  to  realistically  evaluate  cost.  Data  that  is  elicited  using  proven  methods 
and/or  based  on  historical  data  give  more  realistic  and  credible  predictions  than  data  that 
is  simply  a  best  guess.  This  is  where  the  importance  of  data  elicitation  from  the  SME  can 
make  a  dramatic  impact  on  the  quality  of  data  obtained  for  use  in  the  model.  Augustine 
sums  up  the  impact  of  poor  data: 

The  weaker  the  data  available  upon  which  to  base  one’s  conclusion,  the 

greater  the  precision  which  should  be  quoted  in  order  to  give  the  data 

authenticity.  -  Augustine’s  Law  of  Definitive  Imprecision,  Law  Number 

XXXV  (Augustine,  1997). 
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The  final  analysis  of  this  thesis  involves  the  investigation  of  the  impaet  of 
modifying  the  Weibull  distributions  to  simulate  the  MAIMS  principle,  assuming  the 
project  will  spend  all  money  allocated  and  not  have  a  cost  underrun. 

D,  CHAPTER  SUMMARY 

The  NAVSEA  05C’s  cost  analysis  model  of  CG(X)  encompasses  all  aspects  of 
cost  for  the  entire  fleet  including  inflation  and  profit  in  a  63  worksheet  Excel®  workbook. 
The  data  used  in  the  NAVSEA  model  were  acquired  from  SME  inquiry  using  three-point 
estimates  of  high,  most  likely,  and  low  values.  The  Navy  is  placing  great  emphasis  on 
producing  the  best  product  for  each  dollar  spent.  In  order  to  ensure  the  continued 
acquisition  of  CG(X),  it  is  important  that  realistic  cost  risk  analysis  be  performed  so  that 
program  managers  can  make  informed  decisions. 

Chapter  IV  presents  an  approach  to  improve  on  the  model  that  NAVSEA  05  C  has 
provided  for  CG(X).  The  focus  is  strictly  on  the  methodology  used  in  the  cost  analysis  of 
the  electronics  suite  of  CG(X),  represented  in  the  Mission  Systems  Risk  Assessment 
spreadsheet  of  the  model. 
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IV.  REVISED  COST  RISK  ANALYSIS 


A,  INTRODUCTION 

This  thesis  specifically  looks  at  the  electronics  portion  of  the  CG(X)  cost  model 
and  cost  uncertainties  associated  with  engineering  and  manufacturing  of  the  lead  vessel. 
Nine  systems  make  up  the  electronics  suite: 

•  Radar  suite,  which  consists  of  the  following  subsystems:  X-band,  S-band, 
Cooling,  and  Power 

•  ExComm  -  External  Communications 

•  TSCE  -  Total  Ship  Computing  Environment 

•  lUSW  -  Integrated  Undersea  Warfare 

•  EW-IW  -  Electronic  Warfare -Information  Warfare 

•  EO-IR  -  Electro  Optical-Infrared 

•  lEE  -  Identification,  Eriend  or  Eoe 

•  MS  EI&T  (SS  Only)  -  Mission  Systems  Engineering,  Integration,  and 
Testing  (Ship  Systems  Only) 

•  MS  EI&T  (CS  Only)  -  Mission  Systems  Engineering,  Integration,  and 
Testing  (Combat  Systems  Only) 

The  electronics  suite  cost  is  determined  with  the  following  two  equations  that 
treat  each  of  the  cost  elements  as  a  random  variable  (RV).  The  costs  in  bold  represent 
composite  of  the  costs  in  regular  font,  which  Crystal  Ball  refers  to  as  forecasts 
and  assumptions: 

COST(Electronics  Suite)  =  COST(Radar  Suite)  +  COST(ExComm)  + 

COST(TSCE)  +  COST(IUSW)  +  COST(EW-IW)  +  COST(EO-IR)  +  COST(IEE) 

+  COST(MS  EI&T  (SS  Only))  +  COST(MS  EI&T  (CS  Only)), 
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Cost(Radar  Suite)  =  Cost(X-band)  +  Cost(S-band)  +  Cost(Cooling)  + 

Cost(Power). 

The  steps  of  analysis  for  the  CG(X)  model  are: 

1.  Analyze  the  cost  factors  used  by  NAVSEA  05C  to  develop  the  electronics 
suite  cost. 

2.  Analyze  the  PDFs  used  for  the  electronics  cost  elements. 

3.  Identify  what  data  elicitation  methods  were  employed. 

4.  Determine  if  correlation  factors  were  used  in  the  cost  analysis. 

5.  Develop  cost  factors  to  be  modeled  for  cost  realism. 

6.  Decide  which  PDFs  to  use  for  greater  fidelity. 

7.  Develop  an  improved  cost  risk  model  that  includes  realistic  correlation 
factors;  credible  PDFs,  including  MAIMS  influences;  and  SME  biases. 

B,  REVIEW  DEVELOPMENT  OF  COST  FACTORS 

The  cost  factors  used  in  the  NAVSEA05C  CG(X)  cost  model  for  the  electronics 
suite  in  this  thesis  include  data  elicitation  methods,  PDF  choice,  correlation  effects,  and 
MAIMS  Principle  effects.  This  research  does  not  pursue  the  other  elements  of  cost  such 
as  labor,  material,  or  inflation  rates  that  NAVSEA  05C  models. 

1.  Data  Elicitation  Methods 

The  data  elicitation  methods  used  by  NAVSEA  05  C  cost  analysts  are  not  well 
documented.  It  is  clear  that  the  engineering  and  expert  judgment  of  SMEs  is  heavily 
relied  on  for  the  assessment  of  uncertain  cost  elements  associated  with  new  designs.  This 
is  an  area  where  the  use  of  improved  methods  can  dramatically  improve  the  quality  of 
data  that  is  used  in  the  computation  of  the  cost  risk  model.  Subjective  assessments  to 
obtain  data  have  been  identified  as  a  critical  source  of  uncertainty  in  probabilistic  risk 
analyses  (Keeney  &  von  Winterfeld,  1991).  Kujawski  et  al.  (2004)  discuss  the  use  of  the 
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DFA  method  for  data  elicitation  and  how  this  ties  in  with  distribution  choice,  to  provide 
the  most  realistic  cost  assessment. 

DFA  has  been  found  to  provide  one  of  the  most  consistent  and  least  bias-prone 
methods  for  eliciting  uncertain  quantities  (Alpert  &  Raiffa,  1982).  In  their  research, 
people  were  asked  to  consider  uncertain  quantities  by  providing  values  in  terms  of 
percentiles  or  fractiles.  The  findings  indicated: 

•  There  is  a  systematic  bias  toward  overconfidence  in  estimates.  The 
subjective  probability  distributions  were  too  narrow.  Usually,  33% 
instead  of  50%  of  the  actual  values  fell  within  the  0.25  to  0.75  fractiles. 

•  Extreme  value  judgment  is  even  worse.  Twenty  percent,  rather  than  2%, 
of  the  actual  values  fell  outside  the  0.01  and  0.99  fractiles. 

•  What  is  the  meaning  of  minimum  and  maximum  values?  Defining  these 
terms,  so  that  they  are  universal,  is  difficult. 

Kujawski  et  al.  (2004)  further  suggest  using  experts  to  provide  the  10th,  50th,  and 
90th  percentile  values,  as  these  may  be  easier  to  assess  than  extreme  values  of  maximum 
and  minimum.  They  recommend  avoiding  asking  for  extreme  values,  abstract  values 
such  as  the  mean  or  standard  deviation,  or  other  specific  distribution  functions.  If  the 
analyst  does  not  fully  understand  the  background  of  the  questions  being  asked  to  obtain 
data,  or  if  he  does  not  fully  understand  the  behavior  of  the  system  and  associated  data, 
obtaining  discrete  values  will  be  near  impossible. 

Education  also  plays  an  important  role  in  the  quality  of  the  data  provided  by  the 
SMEs  for  analysis.  The  understanding  of  bias  and  its  role  in  affecting  data  elicitation  is 
important.  In  a  presentation  to  the  Navy  Cost  Analysis  Symposium,  Eields  and  Popp 
(2007)  stress  the  importance  of  several  lessons  learned  on  risk.  One  of  the  most 
interesting  of  these  lessons  learned  is  the  importance  of  training.  They  indicate  that 
although  NAVSEA  and  its  technical  community  have  a  broad  cross  section  of 
educational  backgrounds  and  experience,  this  does  not  mean  that  everyone  has 
experience  in  simulation  and  statistics.  The  SME  for  a  particular  electronics  suite 
component  is  probably  not  an  expert  in  probability  and  statistics,  and  because  of  this. 
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tends  to  give  biased  answers  to  the  eost  analysis.  The  distributions  formed  from  the 
biased  data  have  been  found  to  be  partieularly  narrow  and  eentered  on  a  given  point 
estimate,  while  the  extreme  values  are  very  rarely  taken  into  aeeount,  for  reasons 
deseribed  above. 

Edueation  of  the  SMEs  while  eondueting  data  elieitation  is  important,  so  that  the 
experts  have  a  better  understanding  of  what  data  is  required  and  how  it  is  going  to  be 
utilized.  This  training  needs  to  be  eontinually  refreshed  due  to  the  high  turnover  rate  of 
personnel,  whether  they  be  military  or  eivilian,  and  also  beeause  of  improving  methods 
for  eost  analysis.  An  adequate  training  plan  for  both  the  eost  analysts  and  the  SMEs 
providing  data  will  ultimately  result  in  better  data  aequisition  for  eost  analysis. 

In  this  thesis,  the  use  of  DEA  is  simulated  though  the  use  of  Weibull  probability 
distributions  beeause  no  new  data  elieitation  was  eondueted.  The  differenees  between  the 
distributions  using  identieal  values  for  10th,  50th,  and  90th  pereentiles  versus  the  20th, 
50th,  and  80th  pereentiles  illustrates  data  elieitation  that  is  optimistie  versus  pessimistie. 
The  resulting  eost  assoeiated  with  eaeh  of  the  two  distributions  shows  how  dramatie  the 
effeets  of  slightly  different  parameters  ean  have  on  the  estimated  eost. 

2,  Choice  and  Development  of  Probability  Distribution  Functions 

Kujawski  et  al.  (2004)  emphasize  the  importanee  of  realistieally  modeling  eost 
uneertainties  through  the  appropriate  ehoiee  of  probability  distribution  by  meeting  the 
following  eriteria: 

•  Capable  of  fitting  three  arbitrary  pereentiles. 

•  A  finite  lower  range. 

•  An  infinite  upper  range  with  reasonable  behavior. 

•  Physieally  meaningful  and  easy  to  estimate  parameters. 

Three  types  of  PDEs  are  developed  and  modeled  for  this  thesis,  with  the  goal  of 
finding  a  realistie  and  flexible  probability  distribution.  Uneertainty  for  eaeh  eost  element 
in  the  eost  model  is  represented  using  the  same  type  of  PDE  with  different  parameters 
(based  on  NAVSEA  data).  Eirst,  a  triangular  PDE  that  uses  low,  most  likely,  and  high 
values  for  its  parameters  is  developed.  A  lognormal  PDE  that  uses  the  mean  and  standard 
deviation  from  the  triangular  PDE  as  its  parameters  is  the  seeond  distribution.  The  third 
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PDF  is  a  three-parameter  Weibull  distribution  based  on  the  low,  most  likely,  and  high 
values  of  the  triangular  PDF  provided  by  NAVSEA  05  C.  The  low  and  high  values  are 
ealeulated  by  multiplying  the  low  and  high  pereentages  obtained  with  the  most  likely 
value  shown  in  Figure  6.  Two  Weibull  distributions  are  ereated.  One  of  the  Weibull 
distributions  uses  the  data  as  input  for  the  10th,  50th,  and  90th,  pereentiles,  while  the 
other  is  more  pessimistie  and  uses  these  values  for  the  20th,  50th,  and  80th  pereentiles. 
The  data  for  these  parameters  is  taken  from  the  Mission  Systems  Risk  Assessment 
worksheet  (see  Figure  6).  For  eonsisteney,  the  triangular  distribution  is  used  to  determine 
the  50th  pereentile.  The  low,  50th  pereentile,  and  high  values  are  substituted  for  the  20th, 
50th,  and  80th  pereentiles  and  for  the  10th,  50th,  and  90th,  for  eaeh  three-parameter 
Weibull  distribution,  respeetively.  Table  2  shows  the  values  that  were  used  for  the 
parameters  of  eaeh  probability  distribution  funetion  for  the  different  eleetronies 
suite  elements.  Crystal  Ball®  determines  the  three-parameter  Weibull  distribution  by 
using  the  speeified  three  pereentiles.  For  example,  the  parameters  of  the  Weibull  (10%, 
50%,  90%)  distribution  for  the  X-Band  element  are;  loeation  =  60.02,  seale  =  9.21,  and 
shape  =  1.61.  For  thoroughness,  I  verified  that  Crystal  Ball®  generates  identieal 
distributions  using  three  pereentiles  and  the  three  standard  Weibull  parameters. 


A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

1 

Distributions 

2 

Triangular 

Lognormal  Triangular 

Weibull  (10,  SO,  90) 

Weibull  (20,  SO,  80) 

3 

WBS 

Low 

Most  Likely 

High 

Mean 

Standard 

Deviation 

10% 

50% 

90% 

20% 

50% 

80% 

4 

X-band 

S62  30 

565  58 

57541 

$67.76 

$2.78 

562  30 

$67.36 

57541 

562.30 

$67.36 

57541 

5 

S-band 

S146  88 

5154.61 

5177,80 

$159.83 

$6.56 

5146  88 

$158.90 

5177  80 

5146  88 

$158.90 

5177  80 

6 

Cooling 

S3.53 

53.71 

54.27 

$3.84 

$0.16 

53,53 

$3.82 

54.27 

53.53 

$3.82 

54.27 

7 

Power  (included  in  Ship  Total) 

S46.17 

548  60 

555  89 

$50.24 

$2.06 

546  17 

$49.95 

555  89 

546  17 

$49.95 

555  89 

8 

ExComm 

S100.72 

5134.29 

5188  01 

$141.19 

$18.12 

5100.72 

$139.89 

5188  01 

5100.72 

$139.89 

5188.01 

9 

TSCE 

S6649 

575.32 

594  15 

$75.29 

$7.67 

55649 

$75.33 

594  15 

55649 

$75.33 

594  15 

10 

lUSW 

S25  09 

52641 

527  73 

$26.42 

$0.54 

525.09 

$26.41 

527  73 

525  09 

$26.41 

527  73 

11 

EW-IW 

S36  58 

548  78 

560  97 

$48.68 

$5.03 

536  58 

$48.63 

560  97 

536  58 

$48.63 

560  97 

12 

EO-IR 

S1169 

51231 

512.92 

$12.31 

$0.25 

51169 

$12.31 

512.92 

51169 

$12.31 

512.92 

13 

IFF 

S7  98 

5840 

58  82 

$8.40 

$0.17 

57  98 

$8.40 

58  82 

57  98 

$8.40 

58  82 

14 

MS  EI&T(SS  Only) 

S7318 

59147 

5109  77 

$91.61 

$7.55 

573  18 

$78.85 

5109  77 

573  18 

$78.85 

5109  77 

15 

MS  EI&T(CS  Only) 

562,97 

578.71 

594.45 

$78.77 

$6.45 

562.97 

$91.67 

59445 

562.97 

$91.67 

59445 

Table  2.  Parameters  used  in  the  probability  distribution  funetions  for  the  different 

eleetronies  suite  elements. 
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a.  Triangular  Probability  Distribution  Function 

The  parameters  used  to  develop  the  triangular  PDF  are  the  low,  most 
likely,  and  high  values  from  the  Mission  Systems  Risk  Assessment  worksheet  (Figure  6). 
The  determination  of  the  high  and  low  pereentages  for  cost  values  in  the  Mission  Systems 
Risk  Assessment  worksheet  were  figures  given  to  NAVSEA  05  C  cost  analysts  by  SMEs 
from  the  NSWC  Dahlgren.  These  percentages  are  based  on  historical  database  values 
and  inquiry  of  the  SME  for  an  opinion  about  what  they  thought  the  low  and  high  values 
would  be,  based  on  the  most  likely  values  obtained  from  the  historical  databases.  In  this 
case,  data  elicitation  plays  a  big  part  in  the  reliability  of  the  data  used  in  the  model,  which 
is  be  described  in  more  depth  in  Section  IV.B.l. 

The  triangular  distribution  is  not  a  good  predictor  of  high  and  low  costs 
because  it  uses  the  low  and  high  values  as  extreme  values  for  the  end  points.  There  is  no 
allowance  for  costs  above  or  below  the  input  values.  It  has  been  argued  that  a  triangular 
distribution  can  lead  to  either  underestimates  or  overestimates.  Graves  (2001)  states  that 
underestimates  are  likely  due  to  the  finite  upper  limit  of  the  distribution.  Moran  (1999) 
believes  that  overestimates  happen  because  of  the  distribution’s  inability  to  portray  the 
expert’s  confidence  level  of  achieving  the  most  likely  value  and/or  knowledge  of  the 
shape  of  the  distribution.  The  triangular  distribution  is  assigned  a  very  low  score  for 
criteria  (i)  and  (iii)  (Kujawski  et  ah,  2004)  and  is  not  the  chosen  distribution  to  represent 
cost  in  the  model  for  this  thesis. 

b.  Lognormal  Probability  Distribution  Function 

The  lognormal  PDE  is  created  with  the  mean  and  standard  deviation 
parameters  taken  from  the  triangular  distribution.  Characteristics  of  a  lognormal 
distribution  include  being  positively  skewed  with  a  limitless  upper  bound  and  known 
lower  bound.  This  distribution  is  assigned  an  acceptable  score  for  criteria  (iii),  but  a  low 
score  for  (i),  due  to  the  always  positively  skewed  nature  of  the  distribution.  The 
lognormal  distribution  results  in  a  cost  profile  that  closely  follows  with  the  triangular 
distribution,  and  is  one  of  the  narrowest  profiles  modeled.  A  lognormal  PDE  has  been 
associated  with  providing  unreasonably  high  probabilities  at  high  values,  due  to  the 


32 


relatively  slow  falloff  to  the  right.  For  this  reason,  it  gets  an  aeeep table  seore  for  the 
eriteria  (ill),  but  seores  low  on  the  eriteria  (i)  beeause  of  its  always  positively  skewed 
eharaeteristie  (Kujawski  et  ah,  2004). 

c.  Weibull  Probability  Distribution  Function  (10%,  50%,  90%) 

The  three-parameter  Weibull  distribution  is  eharaeterized  by  being  flexible 
and  able  to  assume  a  wide  variety  of  shapes,  while  also  being  open-ended.  Beeause  of  its 
flexible  profile  and  ability  to  mimie  other  distributions,  it  seores  high  on  all  eriteria. 

This  thesis  models  one  of  the  three-parameter  Weibull  distributions  with 
the  10th,  50th,  and  90th  pereentile  values  for  eost.  The  parameters  of  10th,  50th,  and 
90th  pereentiles  are  ehosen  to  simulate  a  eost  environment  that  allows  for  some  eost 
flexibility  on  the  upper  and  lower  limits  rather  than  making  them  extreme  as  in  the 
triangular  distribution.  Although  this  10%  ehange  on  either  side  of  the  distribution  seems 
large,  it  aetually  represents  a  fairly  optimistie  assessment  of  eost.  This  model  is  best  for  a 
situation  in  whieh  the  data  obtained  for  the  model  is  very  reliable. 

d.  Weibull  Probability  Distribution  Function  (20%,  50%,  80%) 

The  three-parameter  Weibull  distribution  using  the  20th,  50th,  and  80th 
pereentiles  for  distribution  parameters  is  intended  to  eorreet  or  aeeount  for  the  overly 
optimistie  biases  diseussed  in  Seetion  2  above.  Systems  that  are  new  and  untested  have  a 
eertain  amount  of  uneertainty  inherent  in  their  aequisition,  and  most  eost  assessments 
made  on  their  eomponents  are  based  on  past  history  if  eomponents  are  being  reprodueed, 
or  a  best  estimate  for  new  systems  and  their  eomponents.  SMEs  are  naturally  optimistie 
about  their  systems  and  have  been  shown  to  give  eost  estimates  that  are  overeonfident, 
resulting  in  probability  distributions  that  do  not  aeeurately  refleet  the  possible  range  of 
eosts  (Kujawski  et  ah,  2004). 

Mueh  data  for  the  CG(X)  eleetronies  suite  is  the  result  of  inquiry  of  SMEs 
and  beeause  of  this,  the  Weibull  distribution  using  20%,  50%,  and  80%  parameters  is 
ehosen  to  model  eosts  for  the  eleetronies  suite  eomponents  in  this  thesis. 
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e.  Cost  Comparisons  with  the  Different  Probability  Distributions 

Figure  9  is  the  Excel®  overlay  created  with  Crystal  Ball®  that  shows  of  a 
10000-run  Monte  Carlo  simulation  for  the  triangular,  lognormal,  Weibull  (10%,  50%, 
90%)  and  Weibull  (20%,  50%,  80%)  distributions,  representing  the  electronics  suite  cost 
of  the  CG(X).  Figure  10  is  the  cumulative  probability  distribution  derived  from  the  PDF 
shown  in  Figure  9.  The  triangular  and  lognormal  distributions  are  very  similar  in  both 
the  probability  distribution  and  cumulative  frequency  functions,  which  is  expected.  Since 
the  lognormal  distribution  uses  the  mean  and  standard  deviation  from  the  triangular 
distribution  as  its  parameters,  the  end  result  should  be  very  similar.  Both  the  triangular 
and  lognormal  functions  show  a  very  distinct  peak  and  sharp  falloff  at  both  the  lower  and 
upper  bounds.  This  behavior  does  not  realistically  model  the  electronics  suite  cost 
because  of  the  sharp  peak  with  sharp  falloff. 


Delta  Elicitation  and  Distribtuion  Selection  Effects 
for  CG(X)  Electronics  Cost 


575  660  745  831  916  993 

Cost.  $M 


Figure  9.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs  showing  the  Effects  of 
Distribution  Choices  on  the  Cost  probability  distributions. 
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Data  Elicitation  and  Distribution  Effects 
for  CG(X)  Electronics  Cost 


584  667  751  834  918  993 

Cost,  $M 

Figure  10.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs  showing  the  Effeets  of 
Distribution  Choiees  on  the  eost  Cumulative  Distribution  Funetions. 

The  Weibull  (10%,  50%,  90%)  distribution  shows  a  broader  eost  range  for  the 
given  probability  braekets.  The  tapering  lower  and  upper  bounds  in  eomparison  to  the 
triangular  and  lognormal  distribution  represent  a  more  likely  eost  outcome.  The  Weibull 
(20%,  50%,  80%)  shows  an  even  larger  cost  range,  which  makes  sense  because  this 
distribution  is  supposed  to  model  a  more  pessimistic  view  of  cost.  Both  of  these 
distributions  are  associated  with  higher  costs  as  the  probability  of  the  cost  increases.  It  is 
important  to  note  the  difference  between  the  optimistic  and  pessimistic  Weibull 
distributions  in  Figure  10.  For  each,  the  cost  increases  with  an  increase  in  probability, 
but  it  is  clear  that  the  model’s  results  indicate  that  the  cost  risk  is  significantly  higher 
using  the  pessimistic  Weibull  distribution. 

3,  Correlation  Effects 

Correlation  effects  are  potentially  important  in  modeling  appropriate  cost 
relationships  between  different  elements  of  systems  and  are  not  conducted  enough  in 
current  cost  analysis  models  (Book,  2001).  Trends  with  correlation  tend  to  lean  toward 
perfect  correlation  because  of  simplicity.  Perfect  correlation  does  help  to  widen  the  range 
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of  outputs  in  the  distribution  functions,  but  this  may  not  be  an  accurate  or  reliable 
representation.  Reasonable  correlation  coefficients  may  provide  more  realistic  and 
credible  estimates  of  project  costs,  rather  than  assuming  perfect  or  zero  correlation. 
Assessing  correlation  coefficients  is  a  difficult  problem.  A  need  exists  for  the 
investigation  and  development  of  a  realistic  and  practical  model  to  account  for 
interrelationships  between  cost  elements. 

Two  types  of  correlations  are  modeled  in  this  thesis: 

•  Correlations  among  cost  elements  within  the  radar  suite.  The  radar  suite 
includes  elements  of  X-band,  S-band,  Cooling,  and  Power.  Dependencies 
among  these  components  are  mainly  from  subsystem  characteristics  such 
as  complexity. 

•  Correlations  among  cost  elements  in  the  entire  electronics  suite. 
Dependencies  among  these  cost  elements  occur  from  the  programmatic 
and  organizational  considerations  common  to  all  cost  elements  that  are  a 
part  of  the  same  project  (Kujawski  et  ah,  2004). 

There  are  two  types  of  correlations:  Pearson  and  Spearman.  Pearson  correlation 
coefficient  determines  the  degree  of  linearity  between  two  random  variables,  while 
Spearman  rank  order  correlation  coefficients  measure  monotonicity.  Correlation  among 
cost  elements  in  the  electronics  suite  is  modeled  with  the  use  of  the  Correlation  Matrix 
function  in  Crystal  Ball®.  Crystal  Ball®  uses  rank  correlations  to  correlate  assumptions. 
This  means  that  the  values  are  not  changed,  but  they  are  rearranged  to  produce  the 
desired  correlation.  Rank  correlation  eliminates  the  need  to  explicitly  model  the 
dependence  between  the  cost  elements.  Garvey  (2000)  advocates  the  use  of  Pearson’s 
correlation.  However,  given  the  limited  information,  rank  correlations  offer  the 
advantage  of  accounting  for  correlations  independent  of  explicit  distribution  and 
dependency  models.  The  use  of  Monte  Carlo  simulations  generates  the  full  PDF  rather 
than  simply  expected  value  and  variance. 

This  thesis  uses  three  sets  of  two  correlation  coefficients  to  model  the  correlation 
between  the  radar  suite  elements  and  the  rest  of  the  electronics  suite  components.  The 
first  set  models  the  distributions  with  correlation  coefficients  of  0.5  for  the  radar  suite 
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elements  and  0.5  for  the  entire  eleetronies  suite  elements.  The  seeond  set  of  eorrelation 
coefficients  is  0.5  and  0.2.  The  third  set  uses  correlation  coefficients  of  0.8  and  0.5. 
Figure  11  shows  the  correlation  chart  used  for  the  (0.5,  0.2)  correlation.  The 
corresponding  Weibull  (20%,  50%,  80%)  distribution  model  that  the  correlation  chart  in 
Figure  1 1  is  applied  to  is  shown  in  Figure  12. 

To  further  explain  the  correlation  chart  in  Figure  11,  please  consider  the  cell 
showing  the  correlation  factor  for  the  E7  (Weibull  20,  50,  80)  and  E8  (Weibull  20,  50, 
80)  cost  elements.  As  indicated  by  Eigure  12,  the  cells  E7  and  E8  correspond  to  the 
X-band  and  S-band  systems,  respectively.  Both  are  components  of  the  radar  suite  and 
therefore  are  likely  to  have  strong  dependencies.  Weibull  (20%,  50%,  80%)  denotes  that 
the  elements  are  modeled  using  three-parameter  Weibull  distribution  given  by  the  20th, 
50th,  and  80th  percentiles.  The  correlation  cell  is  determined  by  the  intersection  of  the 
corresponding  row  and  column  for  the  two  elements.  The  value  of  this  cell  is  0.50.  This 
indicates  that  the  correlation  between  the  elements  X-band  and  S-band  of  the  radar  suite 
are  correlated  by  a  factor  of  0.5. 
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Eigure  11.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Correlation  (0.5,  0.2)  chart  for 

Weibull  (20%,  50%,  80%)  distribution  model. 
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A  B  C  D  E  F  G  H 


1 

2 

3 


4 

WBS 

Low 

Most  Likely 

High 

Distribution  Value 

MAIMS  50% 

MAIMS  Mean 

MAIMS  80% 

Forecast 

5 

Electronics 

J 

633.57 

$ 

748.19 

$ 

910.20 

749.223 

808.572 

826.107 

916.469 

749.223 

6 

Radar  Suite 

$ 

190.32 

$ 

223.90 

$ 

257.49 

311.909 

311.909 

314.702 

319.845 

311.909 

7 

X-band 

J 

62.30 

S 

65  58 

S 

75  41 

74.152 

74.152 

74.152 

75.431 

74.152 

8 

S-band 

$ 

146  88 

S 

154  61 

S 

177  80 

184.516 

184.516 

184.516 

184.516 

184.516 

9 

Cooling 

J 

353 

5 

3  71 

s 

4  27 

3.886 

3.886 

4.003 

4.267 

3.886 

10 

Power  (included  in  Ship  Total) 

S 

4617 

S 

48  60 

s 

55  89 

49.355 

49.355 

52.031 

55.632 

49.355 

11 

ExComm 

s 

100  72 

5 

134  29 

s 

188  01 

99.392 

132.510 

147.348 

187.390 

99.392 

12 

TSCE 

$ 

5649 

S 

75.32 

s 

9415 

67.366 

75.895 

75.946 

94.676 

67.366 

13 

lUSW 

s 

25  09 

S 

26  41 

s 

27  73 

26.075 
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Figure  12.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Weibull  (20%,  50%,  80%) 

distribution  model  with  (0.5,  0.2)  eorrelation. 

Figure  13  is  an  overlay  of  the  different  probability  distributions  for  the  eleetronies 
suite  eost,  produeed  by  using  the  following  three  different  eombinations  of  eorrelation 
eoeffieients  for  the  radar  suite  and  eleetronies  suite: 


•  Correlation  eoeffieients  of  0.5  among  the  radar  suite  eomponents  and  0.2 
between  the  different  eleetronie  suite  eomponents. 

•  Correlation  eoeffieients  of  0.5  among  the  radar  suite  eomponents  and  0.5 
between  the  different  eleetronie  suite  components. 

•  Correlation  coefficients  of  0.8  among  the  radar  suite  components  and  0.5 
between  the  different  electronic  suite  components. 

As  discussed  above,  positive  correlations  give  rise  to  broader  distributions,  which 
reflect  higher  uncertainty.  The  no  correlation  PDF  in  Figure  13  is  the  same  as  the  no 
correlation  shown  in  Figure  9.  They  do  not  appear  to  be  the  same  due  to  the  difference  in 
scale  and  also  due  to  the  fact  that  they  are  from  separate  Monte  Carlo  simulations. 
Although  the  Monte  Carlo  simulations  will  give  similar  results  for  each  run,  they  will  not 
be  identical.  Analysis  of  the  effects  of  the  different  correlation  coefficients  is  discussed 
further  in  Section  IV.C. 
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Figure  13.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Eleetronics  Suite  cost 
based  on  different  correlation  effects  in  cumulative  probability  form. 


4,  MAIMS  Principle  Effects 

The  MAIMS  principle  is  modeled  in  this  thesis  by  using  the  three-parameter 
Weibull  (20%,  50%,  80%)  distribution  function  and  predetermined  percentile  points  for 
the  MAIMS  set  points.  By  implementing  MAIMS  into  the  distribution  function,  any 
value  less  than  the  money  allocation  point  is  equal  to  the  money  allocation  value.  The 
percentage  parameters  used  for  MAIMS  are  the  50th  percentile  or  median,  the  mean,  and 
the  80*’^  percentile  funding  levels.  A  spike,  or  delta  function,  is  observed  in  the  MAIMS 
modified  distributions  at  the  money  allocation  points.  These  money  allocation  points 
correspond  to  the  budget  allocated  to  the  WBS  cost  elements  by  the  project  manager. 

The  MAIMS  modified  functions  are  modeled  by  using  the  following  equation; 

If  Distribution  Value  <  X,  thenX,  else  Distribution  Value 
By  using  this  equation,  the  value  of  the  MAIMS  modified  distribution  will  never  be  less 
than  the  value  X. 

Figure  14  is  a  screenshot  of  a  Monte  Carlo  simulation  run  with  the  MAIMS 
principle.  The  formula  bar  shows  the  equation  used  to  simulate  the  MAIMS  principle  for 
cell  G7.  This  equation  was  used  for  the  three  different  MAIMS  scenarios  modeled. 
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Figure  15  shows  the  resulting  PDFs  obtained  from  the  model  in  Figure  14.  Further 
analysis  of  the  effects  of  the  MAIMS  principle  on  cost  is  discussed  more  in  depth  later  in 
this  chapter,  in  Section  1V.D.3. 
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Figure  14.  CG(X)  Crystal  Ball® Analysis,  10000  Runs,  Electronics  Suite  costs  including 

the  MAIMS  principle. 
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Figure  15.  CG(X)  Crystal  Ball® Analysis,  10000  Runs,  Electronics  Suite  costs  including 

the  MAIMS  principle,  PDEs. 
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5. 


Other  Factors 


Uncertainty  ranges  for  the  low,  most  likely,  and  high  values  are  modeled  using 
triangular  distribution  functions  and  a  Monte  Carlo  simulation.  The  purpose  of  doing  this 
simulation  is  to  ensure  that  any  uncertainty  effeets  from  the  low,  most  likely,  and  high 
values  could  be  aecounted  for  in  the  model.  By  doing  this,  the  low,  most  likely,  and  high 
values  become  assumption  cells  as  defined  by  Crystal  Ball®.  These  assumptions  are  used 
for  the  input  parameters  of  the  “Distribution  Value”  cell,  which  gives  the  cost  value  of 
the  partieular  element.  The  results  indieate  that  the  total  cost  distribution  has  values 
essentially  indistinguishable  from  the  original.  This  is  beeause  sampling  the  distributions 
of  the  individual  low,  most  likely,  and  high  values  produces  values  that  are  very  similar 
to  using  the  original  values.  Beeause  the  modeling  did  not  significantly  change  the  result 
of  the  input  parameters  to  the  “Distribution  Cell,”  uneertainty  distribution  functions  for 
the  low,  most  likely,  and  high  values  were  not  used  in  the  model  for  this  thesis. 
Figure  16  shows  the  model  used  for  this  experiment. 
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Figure  16.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Modeling  Low,  Most  Likely,  and 

High  values:  Low  Value  Distribution. 
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c. 


DESCRIPTION  OF  THE  CRYSTAL  BALL  MODEL 


This  section  describes  the  modeling  and  analysis  of  the  cost  factors  discussed  in 
Section  IV.B  using  Crystal  Ball®. 

The  model  for  the  CG(X)  cost  analysis  for  this  thesis  is  developed  by  using  the 
template  provided  by  NAVSEA  05C  for  their  CG(X)  cost  model.  Specifically,  it  utilizes 
the  low,  most  likely,  and  high  cost  values  for  the  triangular  distribution  from  the  Mission 
Systems  Risk  Assessment  worksheet.  The  low,  high,  and  50th  percentile  values  are  used 
as  the  input  parameters  for  the  three-parameter  Weibull  probability  distributions,  Weibull 
(10%,  50%,  90%)  and  Weibull  (20%,  50%,  80%).  The  lognormal  distributions  used  the 
means  and  standard  deviations  of  the  corresponding  triangular  distributions.  These  cost 
element  distributions  were  created  and  simulations  were  run  with  Crystal  Ball®  software. 
Figure  17  is  a  screenshot  of  a  single  step  run  of  a  Monte  Carlo  simulation  for  a  given  set 
of  PDFs. 


WBS 

Low 

Most  Likely 

High 

Distribution  Value 

MAIMS  50% 

MAIMS  Mean 

MAIMS  80% 

Forecast 

Electronics 

$ 

633.57 

$ 

748.19 

$ 

910.20 

743.6651596 

775.150 

771.210 

816.878 

743.6652 

Radar  Suite 

$ 

190.32 

$ 

223.90 

$ 

257.49 

280.4829027 

281.845 

282.157 

292.309 

280.4829 

X-band 

S 

62  30 

S 

65,58 

S 

7541 

67.9865584 

67.987 

67.987 

70.391 

67.98656 

S-band 

S 

146,88 

S 

154  61 

$ 

177,80 

160.0935981 

160.094 

160.094 

165.789 

160.0936 

Cooling 

s 

3,53 

s 

3,71 

$ 

427 

3.765910723 

3.817 

3.839 

3.984 

3.765911 

Power  (included  in  Ship  Total) 

s 

4617 

s 

48  60 

S 

55,89 

48.63683552 

49.949 

50.238 

52.146 

48.63684 

ExComm 

s 

100,72 

s 

134  29 

s 

188,01 

145.9576333 

145.958 

145.958 

157.587 

145.9576 

TSCE 

s 

56  49 

s 

75,32 

s 

94,15 

58.71313762 

75.332 

75.291 

82.153 

58.71314 

lUSW 

$ 

25,09 

s 

2641 

s 

27,73 

27.54032701 

27.540 

27.540 

27.540 

27.54033 

EW-IW 

s 

36,58 

s 

48  78 

s 

60,97 

49.18924453 

49.189 

49.189 

53.234 

49.18924 

EO-IR 

s 

1169 

s 

1231 

s 

1292 

11.94642499 

12.307 

12.305 

12.525 

11.94642 

IFF 

s 

798 

s 

840 

s 

8  82 

8.312572255 

8.395 

8.396 

8.554 

8.312572 

MS  EI&T(SS  Only) 

s 

73,18 

s 

9147 

s 

109  77 

82.91683278 

82.917 

91.606 

84.592 

82.91683 

MS  EI&T(CS  Only) 

s 

62,97 

s 

78,71 

s 

9445 

78.60608439 

91.667 

78.767 

98.383 

78.60608 

Figure  17.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Screenshot  of  CG(X)  single  step 

Monte  Carlo  Simulation. 

The  left  column  of  the  model  shows  the  components  of  the  electronics  suite.  The 
next  three  columns  contain  the  data  that  was  acquired  from  the  NAVSEA  05  C  Excel® 
worksheets.  The  “Distribution  Value”  column  contains  green  boxes,  which  correspond  to 

the  assumptions  for  the  cost  elements.  The  assumption  cells  are  defined  as  probability 
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distributions  and  contain  the  random  variables  for  the  eost  elements.  The  blue  boxes  in 
the  “Distribution  Value”  eolumn  are  the  foreeasts  that  refer  the  sum  of  the  assumption 
boxes  as  follows:  (1)  The  radar  suite  cost  is  the  probabilistic  sum  of  the  X-band  radar, 
S-band  radar,  eooling,  and  power  eosts;  and  (2)  the  electronics  cost  is  the  probabilistie 
sum  of  the  eost  of  the  radar  suite,  the  ExComm,  TCSE,  lUSW,  EW-IW,  EO-IR,  lEE,  and 
MS  EI&T  eosts.  The  MAIMS  eolumns  represent  the  values  for  the  simulation  that 
correspond  to  a  MAIMS  modified  distribution.  The  MAIMS  50%  column  represents  a 
distribution  truncated  at  the  50th  pereentile,  while  the  MAIMS  mean  is  truneated  at  the 
mean  value,  and  the  MAIMS  80%  is  the  distribution  modified  to  start  at  the  80th 
pereentile  value  of  the  baseline  distributions. 

Eigure  18  shows  a  snapshot  of  how  the  distribution  parameters  are  input  into  the 
model  for  a  triangular  distribution.  The  low,  most  likely,  and  high  eost  values  are  entered 
in  the  appropriate  bloeks  in  the  dialog  sereen  ealled  “Define  Assumption”  to  develop  the 
distribution  that  will  be  represented  by  the  appropriate  cell.  In  this  ease,  the  cell  being 
defined  is  E7. 
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Eigure  18.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Sereenshot  of  the 

Development  of  Triangular  Distribution  Eunction. 
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Figures  19,  20,  and  21  are  screenshots  of  how  distributions  are  created  for  the 
lognormal,  Weibull  (10%,  50%,  90%)  and  Weibull  (20%,  50%,  80%)  distributions.  For 
Figure  18,  the  lognormal  distribution,  and  the  input  parameters  of  mean  and  standard 
deviation  are  obtained  from  the  results  of  the  triangular  distribution  function  from 
Figure  17.  The  three-parameter  Weibull  distributions  shown  in  Figures  19  and  20  use  the 
low  values  as  either  the  10th  or  20th  percentiles  and  the  high  values  as  the  80th  or  90th 
percentiles,  depending  on  the  function  and  the  50th  percentile  for  the  third  parameter. 
Note  that  for  Figures  20  and  21,  the  distributions  represented  for  cell  E7  (X-band)  is 
highly  skewed.  This  is  due  to  the  fact  that  the  low  and  most  likely  values  are  very 
similar,  while  the  high  values  are  significantly  higher,  skewing  the  resulting  distribution. 


Figure  19.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Screenshot  of  the  Development 

of  Lognormal  Distribution  Function. 
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Figure  20.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Screenshot  of  the 

Development  of  Weibull  (10%,  50%,  90%)  Distribution  Function. 


Figure  21.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Screenshot  of  Development  of 

Weibull  (20%,  50%,  80%)  Distribution  Function. 

Correlation  distributions  are  created  by  using  the  Correlation  Matrix  function  of 
Crystal  Ball®.  Specific  correlation  coefficients  are  entered  into  the  distribution  functions 
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of  assumption  cells  in  the  dialog  window  under  the  “Correlate”  button.  When  the  model 
is  run  with  correlation  eoeffieients  set  for  speeifie  distribution  funetions,  the  results  are 
now  correlated.  Figure  22  is  a  sereenshot  showing  how  correlation  eoeffieients  are  added 
to  an  assumption  eell. 


Figure  22.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Sereenshot  of  Development  of 

Model  using  Correlation  Coefficients. 


After  the  models  are  developed  and  run  10000  times,  overlays  are  created  to 
graphieally  show  the  results.  These  overlays  are  deseribed  in  more  detail  in 
Section  IV.D. 


D,  RESULTS 


1,  Effects  of  Distribution  Choice  on  Cost  Forecast 

The  first  distributions  modeled  were  the  single  eleetronics  suite  elements  with 
different  distributions.  For  the  purpose  of  this  thesis,  the  element  ExComm  is  chosen  for 
this  explanation.  Figure  23  shows  the  eumulative  frequeney  distribution  of  the  different 
modeled  distributions  for  the  ExComm  element.  The  triangular  and  lognormal 
distributions  show  similar  characteristies,  whieh  is  expeeted  sinee  the  lognormal 
distribution  uses  parameters  taken  from  the  triangular  distribution  (mean  and  standard 
deviations).  Both  Weibull  distributions  show  expeeted  characteristics.  The  Weibull 
(20%,  50%,  80%)  definitely  indieates  a  more  pessimistic  cost  forecast  because  as  the 
cumulative  probability  inereases,  the  eost  inereases  more  signifieantly  than  for  the 
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Weibull  (10%,  50%,  90%)  distribution.  This  overlay  indicates  that  the  choice  of 
distribution  used  in  modeling  does  play  a  significant  part  in  results  obtained  for  cost.  The 
three-parameter  Weibull  distributions  represent  a  more  realistic  cost  outcome  for  high 
risk  components.  Weibull  distributions  allow  for  modeling  of  highly  complex 
distributions  using  DFA,  while  triangular  distributions  have  a  more  restrictive  shape, 
making  it  difficult  to  fit  three  arbitrary  percentiles  for  the  low,  most  likely,  and  high 
values  (Kujawski  et  ah,  2004). 


80  12b  1/1  21b  2b2  SU2 

Cost,  $M 

Figure  23.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics  Suite 
element  ExComm  cumulative  frequency  distributions  for  different 
probability  distributions. 

The  probability  distribution  functions  shown  in  the  overlay  in  Eigure  24  illustrate 
expected  behaviors  for  the  ExComm  PDEs.  Both  the  triangular  and  lognormal 
distributions  are  narrow  because  the  triangular  distribution  upper  and  lower  bounds  do 
not  allow  for  infinite  upper  cost  ranges.  The  sum  of  the  Weibull  (20%,  50%,  80%) 
distributions  shows  a  more  pessimistic  behavior  in  comparison  to  the  sum  of  the  Weibull 
(10%,  50%,  90%)  distributions. 
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80  125  170  216  261  301 

Cost,  $M 


Figure  24.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics  Suite 

element  ExComm  with  different  PDEs. 

Once  all  the  individual  electronics  suite  elements  are  modeled,  they  are  summed 
up  probabilistically  in  the  main  worksheet  in  Excel  to  obtain  the  entire  electronics  suite 
cost.  The  simulation  selects  a  random  value  from  each  of  the  element  distributions  then 
adds  them  to  create  one  data  point  for  the  total  cost.  This  is  repeated  10000  times  to 
create  the  total  cost  distribution.  When  all  the  distribution  functions  (assumption  cells  in 
the  model)  of  the  electronics  suite  elements  are  probabilistically  summed,  the  resulting 
cost  is  illustrated  in  the  overlay  shown  in  Figure  25.  All  four  distributions  have  the 
appearance  of  a  normal  distribution  consistent  with  the  Central  Limit  Theorem 
(Garvey,  2000). 

The  lognormal  and  triangular  distribution  functions  give  rise  to  relatively  narrow 
total  cost  distributions,  consistent  with  the  finite  ranges  of  the  contributing  triangular 
distributions  and  the  modeling  of  the  lognormal  distributions  using  the  corresponding 
mean  and  standard  deviation  values.  The  Weibull  (10%,  50%,  90%)-based  cost 
distribution  shows  narrower  behavior  for  cost  range  than  the  Weibull  (20%,  50%,  80%)- 
based  distribution.  The  Weibull  (20%,  50%,  80%)-based  distribution  allows  for  more 
uncertainty  in  data  elicitation  from  SMEs.  Weibull  distributions  not  only  show  higher 
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probabilities  of  cost  overruns,  but  also  higher  probabilities  of  cost  underruns.  These 
underruns  reflect  the  assumption  of  10%  and  20%  as  the  low  value  parameter  for  the 
distribution,  rather  than  using  it  as  the  minimum  value.  Figure  26  shows  the  same  data  as 
Figure  25,  except  that  it  is  in  the  cumulative  probability  form. 


Data  Elicitation  and  Distribtuion  Selection  Effects 
for  CG(X)  Electronics  Cost 


575  660  74  5  831  916  993 

Cost.  $M 


Figure  25.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics  Suite  Cost 

based  on  different  distribution  selections. 


Data  Elicitation  and  Distribution  Effects 
for  CG(X)  Electronics  Cost 


Figure  26.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics  Suite  cost 
based  on  different  distribution  selections  in  cumulative  probability  form. 
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2. 


Effects  of  Correlation  on  Cost 


As  discussed  in  Section  IV.B.l,  two  types  of  correlations  are  modeled  in  this 
thesis:  (1)  Correlations  among  cost  elements  within  the  radar  suite,  and  (2)  correlations 
among  cost  elements  in  the  entire  electronics  suite.  This  thesis  uses  the  following  three 
different  correlation  coefficient  factors  to  show  the  correlation  between  the  radar  suite 
elements  and  the  rest  of  the  electronics  suite  components: 

•  Radar  suite  correlation  coefficient  =  0.5,  electronics  suite  correlation 
coefficient  =  0.5 

•  Radar  suite  correlation  coefficient  =  0.5,  electronics  suite  correlation 
coefficient  =  0.2 

•  Radar  suite  correlation  coefficient  =  0.8,  electronics  suite  correlation 
coefficient  =  0.5. 

The  choice  to  use  the  values  listed  above  is  to  simulate  an  environment  that  is  not 
a  perfectly  correlated  or  no-correlation  situation.  The  (0.5,  0.5)  correlation  assumes  that 
there  is  an  equal  correlation  relationship  between  the  subcomponents  of  the  radar  suite 
and  the  elements  of  the  electronics  suite.  The  (0.5,  0.2)  correlation  illustrates  the  effects 
of  having  a  stronger  correlation  between  the  elements  of  the  electronics  suite  than 
between  elements  of  the  entire  electronics  suite.  The  (0.8,  0.5)  correlation  show  the 
impact  of  a  stronger  correlation  between  components  of  one  system  than  between 
different  systems.  These  correlation  coefficients  represent  a  limited  set  of  parameters  for 
investigation  in  this  thesis.  Further  research  in  the  determination  of  appropriate 
correlation  coefficients  and  their  effect  is  needed  to  provide  a  more  complete  analysis. 

The  impact  of  correlation  effects  is  seen  in  Figures  27  and  28.  These  overlays 
show  the  different  distributions  that  are  a  result  of  a  10000-run  Monte  Carlo  simulation 
for  the  correlated  distributions  in  cumulative  distribution  form  (Figure  27)  and  PDF  form 
(Figure  28).  The  blue  PDF  is  the  reference  Weibull  (20%,  50%,  80%)  distribution  with 
no  correlation  effects.  This  distribution  has  the  narrowest  cost  range  when  compared 
with  the  correlated  distributions.  The  cost  ranges  of  the  Weibull  distribution  increases  as 
the  correlation  factors  increase.  Also,  in  the  cumulative  probability  distribution  shown  in 

Figure  27,  all  of  the  distributions  intersect  at  the  mean  value  for  the  cost. 
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CGX  Electronics  Cost 
Correlation  Effects 


Figure  27.  Figure  27:  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics 

Suite  Cost  showing  the  impact  of  different  correlation  effects. 


CGX  Electronics  Cost 
Correlation  Effects 


Figure  28.  Figure  28:  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics 
Suite  cost  based  on  different  correlation  effects  in  cumulative  probability  form. 

As  expected,  the  (0.5,  0.2)  correlation  being  the  smallest  has  the  least  effect  on  the 
total  cost  distribution.  It  is  interesting  to  note  that  the  distribution  resulting  from  the 
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(0.5,  0.5)  correlation  does  not  differ  mueh  from  the  (0.8,  0.5)  distribution,  but  that  both  of 
these  eorrelations  have  a  more  signifieant  effeet  than  the  (0.5,  0.2)  eorrelation.  This 
indieates  that  a  ehange  in  the  the  correlation  faetor  for  the  radar  suite  from  0.5  to  0.8  is 
not  as  signifioant  as  a  change  in  the  eorrelation  faetor  from  0.2  to  0.5  for  the  different 
eomponents  of  the  entire  eleetronics  suite.  These  results  suggest  that  the  eorrelation 
effeets  are  important  for  probability  values  midpoint  between  the  mean  and  the  extremes, 
but  there  is  little  differenee  for  values  beyond  0.5.  The  results  in  Figure  28  are  eonsistent 
with  theoretieal  predietions  of  positive  eorrelation  effeets  in  that  the  total  eost  beeomes 
broader  than  for  uneorrelated  total  eost  (Kujawski  et  ah,  2004).  Further  investigations  are 
reeommended  to  quantify  correlation  effeets. 

3.  MAIMS  Effects  on  Cost 

The  MAIMS  modified  eumulative  probability  and  density  density  distributions 
for  the  eleetronics  suite  eost  are  shown  in  Figures  29  and  30.  Charaeteristies  of  the 
MAIMS  modified  PDF  is  that  they  will  never  have  a  value  less  than  the  ehosen  value  of 
modification.  So,  for  the  MAIMS  50th  pereentile  modified  distribution  in  Figure  29,  the 
distribution  has  no  value  less  than  the  50th  pereentile  baseline  eost  level.  In  Figure  30, 
the  spikes  or  delta  funetions  normally  assoeiated  with  the  individual  MAIMS 
distributions  are  not  seen  as  they  are  modulated  when  summed. 
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CGX  Electronics  Cost 
Weibull  (20.  50.  80)  MAIMS  Effects 


Figure  29.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Eleetronics  Suite  cost 

showing  the  MAIMS  effects  in  cumulative  probability  form. 


CGX  Electronics  Cost 
Weibull  (20.  50.  80)  MAIMS  Effects 


637  694  752  809  866  918 


Cost,  $M 


Figure  30.  CG(X)  Crystal  Ball®  Analysis,  10000  Runs,  Overlay  of  Electronics  Suite  cost 

showing  the  MAIMS  effects  in  PDE  form. 
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It  is  important  to  note  the  significant  rise  in  cost  curve  that  occurs  as  the  MAIMS 
modification  value  increases.  The  mean  value  of  the  distribtion  increases  as  the  funding 
level  increases,  and  this  is  very  clear  in  Figure  29.  The  curve  representing  the  MAIMS 
80%  distribution  shows  how  the  budget  is  always  high  when  comparing  it  to  the  MAIMS 
50%  or  MAIMS  mean  distribution.  This  effect  is  because  once  money  has  been  allocated 
to  a  WBS  element,  it  is  almost  never  seen  in  cost  savings  as  underruns,  because  cost 
account  managers  never  return  money  to  the  project.  Any  remaining  money  from  one 
WBS  is  subsequently  spent  on  a  different  existing  WBS  that  has  cost  overruns. 

These  simulations  can  be  considered  with  other  cost  factors  in  making  program 
management  decisions  regarding  budgets.  Funding  projects  at  a  level  too  low  to  cover 
costs  will  lead  to  cost  overruns,  while  funding  at  a  level  that  is  too  high  leads  to  money 
not  being  recouped  as  savings  later.  Allocating  reasonable  budgets  is  the  goal. 

E.  CHAPTER  SUMMARY 

The  research  for  this  thesis  is  based  on  the  NAVSEA  05  C  CG(X)  model  provided 
by  Mr.  Chris  Deegan  and  his  CG(X)  analysts.  The  CG(X)  model  encompasses  all  factors 
considered  for  cost  of  the  entire  program,  including  labor  rates,  material  cost,  overhead 
cost,  planning  cost,  and  other  factors.  Because  of  the  complexity  of  the  model  and  the 
numerous  factors  to  consider,  one  portion  of  the  model  was  chosen  for  analysis.  The 
Electronic  Suite  and  its  nine  elements  are  specifically  targeted  as  the  focus  for  analysis. 

The  steps  used  in  the  analysis  of  the  CG(X)  model  are: 

1.  Analyze  the  cost  factors  used  by  NAVSEA  05C  to  develop  the 
electronics  cost. 

2.  Analyze  the  PDEs  used  for  the  electronics  cost  elements. 

3.  Identify  what  data  elicitation  methods  were  employed. 

4.  Determine  if  correlation  factors  were  used  in  the  cost  analysis. 

5.  Develop  cost  factors  to  be  modeled  in  a  new  model. 

6.  Decide  which  PDEs  to  use  in  the  new  model. 

7.  Develop  a  new  cost  model  using  correlation  factors,  chosen  PDEs,  and 
MAIMS  influenced  distributions. 
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Identified  cost  factors  include  NAVSEA  05C’s  probability  distribution  choice, 
method  used  for  developing  the  low,  most  likely,  and  high  cost  values  for  the  electronics 
suite  elements,  data  elicitation  methods,  and  correlation  effects.  This  thesis  explores  the 
methodology  in  choosing  different  probability  distribution  functions  and  their 
applicability  to  the  model.  Specifically,  triangular,  lognormal,  and  two  variations  of  the 
three-parameter  Weibull  distribution  are  considered. 

Methods  of  data  elicitation  are  explored  and  the  use  of  a  DFA  method  is 
recommended  for  future  use,  although  the  research  in  this  thesis  did  not  involve  data 
acquisition.  To  simulate  the  use  of  a  DFA  methodology,  Weibull  distributions  are 
employed  to  account  for  uncertainty  associated  with  SME  estimation  of  data.  A  Weibull 
(10%,  50%,  90%)  distribution  is  used  to  simulate  a  more  optimistic  view  of  the 
uncertainty  of  data,  while  a  Weibull  (20%,  50%,  80%)  distribution  models  a  more 
pessimistic,  but  probably  more  realistic,  view  of  the  uncertainty  associated  with  data  from 
the  SMEs. 

Two  types  of  correlation  effects  are  considered  and  modeled  in  this  thesis.  The 
first  is  the  correlation  between  subcomponents  of  the  radar  suite  and  the  other  is  the 
correlation  between  the  elements  of  the  electronics  suite.  The  radar  suite  is  one  of  the 
elements  that  make  up  the  electronics  suite.  Analysis  shows  that  a  more  significant  effect 
is  experienced  with  higher  correlation  between  the  elements  of  the  electronics  suite  than 
between  the  subcomponents  of  the  radar  suite. 

MAIMS  modified  probability  distributions  are  modeled  to  show  the  significance 
of  budget  allocation  level.  These  distributions  are  truncated  at  the  baseline  budget  with  a 
delta  function  at  the  baseline.  This  is  based  on  the  principle  that  once  a  budget  is 
allocated,  money  is  almost  never  seen  in  the  form  of  cost  under  runs  as  the  project 
progresses.  As  the  MAIMS  modification  value  increases,  overall  distribution  cost  rises 
with  increasing  probability  of  success. 
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V.  CONCLUSIONS 


A,  SUMMARY 

This  thesis  begins  by  exploring  the  definitions  of  risk  and  how  it  applies  to  the 
guidanee  set  forth  by  eurrent  Navy  leadership.  Admiral  Gary  Roughead,  Chief  of  Naval 
Operations,  states  that,  “We  manage  risk”  (Roughead,  2007).  The  need  to  develop 
effeetive  aequisition  and  shipbuilding  methods  to  suoeessfully  deliver  an  “affordable 
future  fleet”  (McCoy,  2008)  is  imperative  if  the  Navy  is  to  meet  the  goal  of  a  313-ship 
Navy  by  2020.  Cost  risk  analysis  is  one  tool  of  many  that  can  be  used  to  help  attain 
this  goal. 

This  thesis  then  proceeds  to  examine  the  probabilistic  cost  analysis  approach  that 
NAVSEA  05  C  currently  uses  to  predict  new  naval  vessel  construction  costs  and  to 
develop  a  method  that  better  predicts  the  ultimate  cost  risk.  Cost  factors  analyzed  in  this 
thesis  include  the  effect  of  data  elicitation,  distribution  choice,  the  impact  of  the  MAIMS 
principle,  and  the  effect  of  correlation  factors.  Data  elicitation  and  MAIMS  have 
significant  impact.  Correlation  effects  vanish  at  the  minimum,  mean,  and  maximum 
values.  PDF  selection  has  a  small  impact  as  long  as  the  distributions  fit  the  three 
specified  percentiles. 

The  model  provided  by  NAVSEA  05 C  encompasses  all  aspects  of  the  ship’s  cost 
and  only  the  nine  elements  of  the  electronics  suite  were  chosen  for  analysis  in  this  thesis. 
Using  data  obtained  from  SMEs  for  low,  most  likely,  and  high  cost  values,  experiments 
were  conducted  for  the  noted  cost  factors  in  the  Excel®  Monte  Carlo  simulation  add-in 
Crystal  Ball®. 

Triangular,  lognormal,  Weibull  (10%,  50%,  90%)  and  Weibull  (20%,  50%,  80%) 
distributions  are  modeled  and  simulated  to  show  the  impact  that  each  distribution  can 
have  on  budget  considerations  for  program  managers.  Both  the  triangular  and  lognormal 
distributions  show  narrow  cost  ranges  when  compared  to  the  Weibull  distribution  cost 
range.  The  Weibull  (10%,  50%,  90%)  represents  a  more  optimistic  distribution  than  the 
more  pessimistic  Weibull  (20%,  50%,  80%)  distribution.  The  Weibull  (20%,  50%,  80%) 
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distribution  accounts  for  the  optimism  bias  commonly  associated  with  SMEs.  Data 
elieitation  effeets  are  modeled  through  the  use  of  the  Weibull  distributions. 

Correlation  among  eost  elements  in  the  eleetronies  suite  is  modeled  with  the  use 
of  the  Correlation  Matrix  funetion  in  Crystal  Ball®.  This  thesis  uses  three  sets  of  two 
eorrelation  eoeffieients  to  model  the  eorrelation  between  the  radar  suite  elements  and  the 
rest  of  the  eleetronies  suite  eomponents.  The  results  suggest  that  the  eorrelation  effeets 
are  important  for  probability  values  midpoint  between  the  mean  and  the  extremes,  but 
there  is  little  differenee  for  values  beyond  0.5.  Further  investigations  are  reeommended 
to  quantify  eorrelation  effeets. 

MAIMS  prineiple  modified  distributions  are  modeled  with  the  50th  pereentile 
eost  value,  mean,  and  80th  pereentile  eost  value  to  show  the  impaet  of  funding  at  these 
different  levels.  The  MAIMS  prineiple  is  based  the  observation  that  for  a  given  budget, 
any  money  alloeated  is  eonsidered  money  spent.  Very  rarely  are  eost  underruns 
experieneed  on  a  projeet  onee  the  budget  has  been  alloeated.  The  MAIMS  modified 
distributions  in  this  thesis  show  the  impaet  of  either  under-funding  a  budget  or  over- 
funding.  Under-funding  leads  to  eost  overruns  and  over-funding  leads  to  an  overall 
higher  eost,  sinee  money  alloeated  is  unlikely  to  be  reeouped. 

B,  RECOMMENDATIONS  AND  AREAS  FOR  FURTHER  RESEARCH 

The  analysis  eondueted  in  this  model  is  only  a  starting  point  for  improvements  in 
the  area  of  eost  analysis  for  naval  vessels.  Although  the  methodology  used  in  this  thesis 
provides  a  framework  for  obtaining  more  aeeurate  predietions  of  eost  than  those  in  use 
with  eurrent  probabilistie  eost  analysis,  more  work  is  required  to  develop  a  more 
eomplete  and  tested  model.  Reeommendations  for  future  researeh  in  the  area  of 
probabilistie  eost  analysis  for  shipbuilding  inelude: 

•  Use  of  the  DFA  method  to  obtain  data  for  eost  assessment.  Reeommend 
elieiting  data  from  SMEs  at  the  10th,  50th,  and  90th  pereentiles,  at  a 
minimum,  for  relatively  optimistie  view  of  the  data  quality,  and  at  the 
20th,  50th,  and  80th  pereentiles  if  a  more  pessimistie  view  of  the  quality 
of  data  is  present.  Take  into  eonsideration  the  overeonfidenee  of  estimates 
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provided  by  experts  in  their  field  and  use  this  knowledge  when  calibrating 
data  for  analysis. 

•  Select  flexible  and  realistic  probability  distribution  functions  for 
cost  analysis.  Create  probability  distribution  functions  from  historical  data 
and  adjust  for  expected  differences  in  new  programs. 

•  Incorporate  the  use  of  correlation  among  cost  elements  of  a  system.  Aim 
to  use  a  range  of  correlation  coefficients  that  is  realistic.  A  reasonable 
range  for  correlation  coefficients  is  between  0.3-0. 6,  with  some  room  for 
variation.  Overly  optimistic  correlation  coefficients  that  assume 
independence  and  overly  pessimistic  correlation  coefficients  that  assume 
perfect  correlation  rarely  exist  in  real  data. 

•  Use  the  “Money  Allocated  is  Money  Spenf  ’  (MAIMS)  principle  to  model 
budget  management  behavior.  The  MAIMS  function  will  not  allow  the 
system  cost  to  be  a  lesser  amount  than  the  budgeted  cost  baseline. 

•  Investigate  further  capabilities  available  with  advanced  modeling  software 
such  as  Crystal  Ball®  or  @Risk. 

•  Incorporate  systems  engineering  methodologies  and  thinking  into  the 
development  of  probabilistic  cost  analysis.  Kujawski  et  al.  (2004)  state 
that  this  is  the  single  greatest  challenge  to  the  development  and  use  of 
improved  cost  models. 

Continuing  with  the  development  of  improved  cost  models  is  an  important  step  in 
helping  the  Navy  to  ensure  the  successful  acquisition  of  the  313-ship  Navy  it  desires. 
Improved  cost  models  can  give  project  managers  the  ability  to  develop  more  realistic  and 
successful  plans  for  their  projects,  while  enabling  them  to  make  better  budget  decisions. 
The  cost  analysis  methodology  presented  in  this  thesis  can  serve  as  a  starting  point  for 
further  advanced  research  in  this  area  that  can  be  used  by  different  programs  across 
the  Navy. 
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